You are building the eval set for a production LLM system. How do you decide how many examples are enough? What criteria signal that adding more examples will not improve eval quality, and what signals tell you the opposite?
Free · 2 AI evals / day
A **golden eval set** is a curated, stable list of examples you run a model against to measure quality. The instinct is to ask 'how many?' but that is the wrong question. The right one is: what decision will this set make, and how small a difference must it detect?
A common baseline is 100 to 500 well-chosen examples, but that number is downstream of the job. To catch a big 10-point drop, a small set works. To catch a 2-point regression, you need many more examples to see that signal through the noise.
You have enough when two things hold. First, running the same set twice gives nearly the same score, so the run to run wobble is smaller than the difference you care about. Second, every known failure category and important query type has its own examples, not just the overall total.
You need more when a slice, say refund questions, has only a handful of examples, when you find a new failure mode the set never covered, or when scores jump around between identical runs.
The headline mistake is fixating on total count. A stratified set of 300 beats a random pile of 3,000 that misses your hard slices.
Size the set to the decision, not to a folklore number. The real question is the smallest score difference you must reliably detect, the **minimum detectable effect**.
Start from the confidence interval on a pass rate. For a proportion the half-width is:
$$\text{CI}_{95} \approx 1.96 \sqrt{\frac{p(1-p)}{n}}$$
A worst-case rate of 0.5 at n=100 gives a half-width near 10 points. At n=400 it halves to 5. Resolution improves only as the square root of n, so doubling precision costs four times the examples.
Two stopping signals say you have enough. Run to run variance on repeated passes is small relative to your regression threshold. And every failure category and query slice clears its own minimum, roughly 30 examples for a stable rate.
Three signals say you need more. A slice has too few examples to score. You discover a failure mode the set never covered. Or repeated runs wobble more than the difference you act on.
The trap is per-slice resolution. A total of 1,000 examples can still leave a critical slice with only 8. That slice has a confidence interval near 35 points wide. So each slice needs its own n, which is why stratified sizing beats raw count almost every time.
Quick reference
TL;DR
Size to the decision, not a magic number. Pick the smallest n that resolves the score difference you care about, then make each slice clear its own minimum independently.
Imagine deciding whether a new recipe is better by feeding it to taste-testers. With three testers, one grumpy person flips the verdict, so the result is noise. Add more testers and the average steadies, but past a point a new taster barely moves it. The right number is the smallest crowd whose average is steady enough that a real improvement still shows through the wobble. There is a catch. If you want to know how the recipe does for kids versus adults, you cannot just count the whole crowd. Each group needs enough of its own testers, or that group's verdict stays noisy even when the overall crowd is huge. So you size for the question you are actually asking, group by group, not for one big total.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Exact-match tasks with cheap, unambiguous ground truth such as classification or extraction You can run the full population or a very large sample cheaply, so careful sizing math adds little value.
Very early prototyping before the product behavior has stabilized The failure modes are still shifting, so any sized set goes stale fast. Spend effort on coverage breadth, not statistical rigor.
Smoke tests meant only to catch gross breakage, not to measure small quality changes A tiny fixed set is fine when the detectable effect you need is large and obvious.
A single global metric with no meaningful subpopulations to break out Per-slice sizing is moot when there are no slices that gate the decision separately.
Quick scan · 2–4 min
Everything important, quickly.
5 min: derive n from the minimum detectable effect via the proportion CI, explain square-root scaling, enforce per-slice minimums, separate sampling from judge variance, and reason about the cost versus resolution trade.
Real-world examples
Real products, models, and research that use this idea.
1LangSmith and Braintrust report per-slice eval scores with sample counts, surfacing thin slices before you trust a global mean.
2RAGAS faithfulness scores are computed per example so teams can bootstrap a confidence interval on the aggregate rather than read a bare mean.
3Chatbot Arena needs tens of thousands of pairwise votes precisely because resolving small Elo gaps between Claude Opus 4.7 and GPT-5.5 demands tight intervals.
4OpenAI Evals and Promptfoo let you re-run an identical suite to quantify run to run judge variance as a first-class stopping signal.
5Patronus and Galileo stratify production eval sets by query type so each failure category clears its own minimum example count.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you separate true regression from judge stochasticity when scores wobble between runs?
Sign in or go Pro to see the approach for this follow-up.
QYour global score is fine but one slice is only 6 examples wide. How do you decide whether to grow it?
Sign in or go Pro to see the approach for this follow-up.
QWhy does doubling eval precision cost roughly four times the examples, and when is that worth paying?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Citing a fixed number like 500 as if it were universal, instead of deriving n from the smallest score difference you must detect and the confidence interval width your decision can tolerate.