How large should a golden eval set be, and what signals tell you when you have enough examples?
You are building the eval set for a production LLM system. How do you decide how many examples are enough? What criteria signal that adding more examples will not improve eval quality, and what signals tell you the opposite?
Size to the decision, not a magic number. Pick the smallest n that resolves the score difference you care about, then make each slice clear its own minimum independently.
Imagine deciding whether a new recipe is better by feeding it to taste-testers. With three testers, one grumpy person flips the verdict, so the result is noise. Add more testers and the average steadies, but past a point a new taster barely moves it. The right number is the smallest crowd whose average is steady enough that a real improvement still shows through the wobble. There is a catch. If you want to know how the recipe does for kids versus adults, you cannot just count the whole crowd. Each group needs enough of its own testers, or that group's verdict stays noisy even when the overall crowd is huge. So you size for the question you are asking, group by group, not for one big total.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Golden set sizing is one of the few LLM-eval topics with a real statistical core. Reciting a number such as 'use 100 to 500 examples' skips that core. A sound approach treats the eval set as a measuring instrument and derives its size from the resolution the instrument must deliver.
The central idea is that an eval set exists to make a decision: ship or hold, model A or model B, regression or noise. Each decision has a smallest quality difference that must be detected reliably, the minimum detectable effect. Once you fix that, the sample size follows from basic sampling theory. The same math also explains the most expensive mistake: sizing the global total while leaving individual slices with too few examples.
Start from the minimum detectable effect
Before touching a number, write down the decision the eval gates and the smallest score change that should flip it. Catching a catastrophic 20-point drop is a different instrument than catching a 2-point regression between two strong models. The smaller the effect you must resolve, the more examples you need.
This reframing rules out a fixed rule-of-thumb number. 'Use 500' is meaningless without the effect size. Five hundred examples that resolve 5 points might be perfect for a coarse gate. They might also be hopelessly underpowered for a tight model-selection call where the candidates sit 2 points apart.
Fixing the minimum detectable effect also forces an honest conversation about error rates. A test with too few examples has low power, so it misses real regressions and ships them silently. A test sized far past what the decision needs spends judge budget detecting differences nobody will ever act on. The minimum detectable effect sets the balance between those two failure modes. The person who owns the ship decision should set it. It should not come by default from whatever number of examples happened to be on hand.
The minimum detectable effect is a product decision, not a statistics decision. The engineer's job is to translate it into a sample size, then report honestly what resolution the chosen size delivers. Sometimes the requested effect is smaller than the budget can resolve. Raise that conflict early rather than shipping an underpowered gate that looks rigorous on a dashboard.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith and Braintrust report per-slice eval scores with sample counts, surfacing thin slices before you trust a global mean.
- RAGAS faithfulness scores are computed per example so teams can bootstrap a confidence interval on the aggregate rather than read a bare mean.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you separate true regression from judge stochasticity when scores wobble between runs?
Re-run the identical set multiple times at temperature zero with a pinned judge version to estimate the irreducible judge variance floor. Only a between-version gap that exceeds that floor and the sampling interval counts as a real regression. Bootstrap the difference and check the confidence interval excludes zero.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Citing a fixed number like 500 as if it were universal. Instead, derive n from the smallest score difference you must detect and the confidence interval width your decision can tolerate.
60 second bullets to scan on the way to the call.
How a proportion confidence interval width depends on sample size
Why resolution scales as one over the square root of n
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.