Predict the number of LLM calls this Promptfoo matrix configuration will issue
A promptfooconfig.yaml declares 3 prompts, 4 providers, and 25 test cases. defaultTest.options.repeats is set to 2. No provider specific tests are skipped. Predict the total number of LLM completion calls Promptfoo will make for one full run, ignoring any rate limit retries.
Promptfoo runs the full Cartesian product: prompts × providers × tests × repeats = 3 × 4 × 25 × 2 = 600 LLM completion calls.
Think of a chef testing three sauce recipes on four pans with twenty-five different cuts of meat, and doing each combination twice to make sure the result is consistent. You write down every combination you tried. Count them: three sauces, times four pans, times twenty-five cuts, times two tries, equals six hundred test plates. Nothing is skipped unless you tell the chef in advance to skip a specific pairing. Repeats is the just to be sure pass, and it multiplies everything because every combination is tried twice.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Promptfoo's evaluation matrix is the Cartesian product of every dimension declared in promptfooconfig.yaml. Most teams understand this in the abstract but forget that every dimension is a multiplier, and the multipliers compound silently. The result is eval-CI bills that surprise the team three months after adoption.
This walkthrough computes the answer for the scenario (600 calls), generalizes the formula, lays out the cost and time implications at production scale, names the knobs that scale cost dramatically, and walks the operational guardrails that keep eval-CI within budget.
Mental model: the YAML declares dimensions; the run is the Cartesian product times the repeats multiplier. Adding any dimension scales every other dimension.
Computing the answer and generalizing the formula
The scenario answer
Given:
- 3 prompts
- 4 providers
- 25 test cases
- 2 repeats per cell
- No provider-specific test exclusions
The matrix is the Cartesian product times the repeats:
total_calls = prompts × providers × tests × repeats = 3 × 4 × 25 × 2 = 600
Generalizing
The formula extends to any combination Promptfoo supports:
total_calls = (prompts) × (providers) × (tests) × (repeats) × (variants per test)
If the YAML declares prompt variants under each prompt (e.g., 2 variants per prompt), the prompt dimension becomes prompts × variants. Same for any other multiplicative dimension Promptfoo adds in future versions.
Provider-specific test exclusions subtract from the dense product. If 5 of 25 tests are limited to a single provider, the matrix shrinks by 5 × (providers - 1) × prompts × repeats = 5 × 3 × 3 × 2 = 90 calls, giving 510 instead of 600.
Why each dimension matters
- Prompts capture prompt-engineering variants under comparison.
- Providers capture model/vendor comparisons.
- Tests capture the golden-set coverage.
- Repeats captures noise-robustness for stochastic outputs (where the same input yields different outputs across runs).
None are wasted; each answers a real question. But every one is a multiplier on every other one.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Promptfoo's own documentation on matrix evaluation shows the Cartesian-product pattern with worked examples.
- Notion's prompt-engineering team uses Promptfoo in CI with a pre-run matrix-size guard to cap any single PR's eval cost.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design a sampled Promptfoo matrix for PR-level CI versus a full matrix for merge to main?
PR-level: sample 5 to 10 representative tests per provider, use repeats=1, total ~100 calls. Merge to main: full matrix with repeats=2 or 3 for confidence intervals. Different config files or a flag-controlled subset. Sampling strategy should be stratified across test categories so coverage is preserved.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that repeats multiplies every cell rather than running once at the end. With repeats=2 the entire matrix runs twice; with repeats=5 it runs five times. Misreading this is how teams accidentally burn five-figure eval budgets in a single CI run.
60 second bullets to scan on the way to the call.
The Cartesian-product formula: prompts × providers × tests × repeats
Why repeats is a multiplier on every cell rather than an additive pass
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.