How large should a golden eval set be, and what signals tell you when you have enough examples?
You are building the eval set for a production LLM system. How do you decide how many examples are enough? What criteria signal that adding more examples will not improve eval quality, and what signals tell you the opposite?
Size to the decision, not a magic number. Pick the smallest n that resolves the score difference you care about, then make each slice clear its own minimum independently.
Imagine deciding whether a new recipe is better by feeding it to taste-testers. With three testers, one grumpy person flips the verdict, so the result is noise. Add more testers and the average steadies, but past a point a new taster barely moves it. The right number is the smallest crowd whose average is steady enough that a real improvement still shows through the wobble. There is a catch. If you want to know how the recipe does for kids versus adults, you cannot just count the whole crowd. Each group needs enough of its own testers, or that group's verdict stays noisy even when the overall crowd is huge. So you size for the question you are actually asking, group by group, not for one big total.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Golden set sizing is one of the few LLM-eval questions with a real statistical core, which is why it separates senior from staff answers. The weak answer recites a number: 'use 100 to 500 examples.' The strong answer treats the eval set as a measuring instrument and derives its size from the resolution the instrument must deliver.
The central idea is that an eval set exists to make a decision: ship or hold, model A or model B, regression or noise. Each decision has a smallest quality difference that must be detected reliably, the minimum detectable effect. Once you fix that, the sample size follows from basic sampling theory. And the most expensive mistake, sizing the global total while starving individual slices, falls straight out of the same math.
The rest of this deep dive builds the argument in five moves: anchor on the minimum detectable effect, derive n from the proportion confidence interval, show why per-slice minimums make the grand total a derived quantity, separate the two kinds of variance so you do not chase noise, and close on the cost-versus-resolution trade that governs when to stop. A worked regression gate ties all five together with concrete numbers.
Start from the minimum detectable effect
Before touching a number, write down the decision the eval gates and the smallest score change that should flip it. Catching a catastrophic 20-point drop is a different instrument than catching a 2-point regression between two strong models. The smaller the effect you must resolve, the more examples you need.
This reframing matters because it kills the folklore answer. 'Use 500' is meaningless without the effect size. Five hundred examples that resolve 5 points might be perfect for a coarse gate and hopelessly underpowered for a tight model-selection call where the candidates sit 2 points apart.
Fixing the minimum detectable effect also forces an honest conversation about error rates. A test with too few examples has low power, so it misses real regressions and ships them silently. A test sized far past what the decision needs spends judge budget detecting differences nobody will ever act on. The minimum detectable effect is the dial that balances those two failure modes, and it should be set by whoever owns the ship decision, not chosen implicitly by whatever number of examples happened to be lying around.
The minimum detectable effect is a product decision, not a statistics decision. The engineer's job is to translate it into a sample size, then report honestly what resolution the chosen size actually delivers. When the requested effect is smaller than the budget can resolve, surface that tension early rather than quietly shipping an underpowered gate that looks rigorous on a dashboard.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith and Braintrust report per-slice eval scores with sample counts, surfacing thin slices before you trust a global mean.
- RAGAS faithfulness scores are computed per example so teams can bootstrap a confidence interval on the aggregate rather than read a bare mean.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you separate true regression from judge stochasticity when scores wobble between runs?
Re-run the identical set multiple times at temperature zero with a pinned judge version to estimate the irreducible judge variance floor. Only a between-version gap that exceeds that floor and the sampling interval counts as a real regression. Bootstrap the difference and check the confidence interval excludes zero.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Citing a fixed number like 500 as if it were universal, instead of deriving n from the smallest score difference you must detect and the confidence interval width your decision can tolerate.
60 second bullets to scan on the way to the call.
How a proportion confidence interval width depends on sample size
Why resolution scales as one over the square root of n
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.