What is the practical guidance on golden eval set size for LLM systems?
Golden set size is driven by the effect size you must detect and per-slice coverage, not a round number. 100 to 500 well-stratified examples usually suffice; the pass-rate confidence interval tells you if you need more.
Imagine polling whether a town likes a new park. You do not poll everyone. A few hundred well-chosen people give a tight enough estimate. But if you want to know what each neighborhood thinks separately, each neighborhood needs its own sample, so the total grows. An eval set works the same way. A few hundred examples pins down your overall pass rate to within a few points. The catch: if you care about each customer type or each failure mode separately, each slice needs enough examples on its own. And if you re-run the same set and the score jumps around, adding examples will not save you. You first have to make the measurement stable, by pinning the judge to a fixed temperature.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
"How big should my golden eval set be?" is one of the most common LLM-evaluation interview questions, and the weak answer is a round number. The strong answer reframes it as a measurement-design question: how small a quality change must you reliably detect, and how many slices must you report separately?
The headline metric of most LLM evals is a pass rate or win rate, which is just a proportion. That single fact gives you the whole statistical backbone. The width of the confidence interval on a proportion tells you how precise your estimate is, and that width is what should drive the size, not folklore. The familiar 100-to-500 guidance is simply what falls out when you combine a moderate target effect size with a handful of reported slices.
The walk-through below works through the proportion relation that sets the floor, the effect-size logic that turns it into a real number, the way stratification multiplies that number, and the variance trap that quietly defeats sizing altogether. By the end you should be able to defend a specific number rather than reach for a habit.
Why a pass rate is a proportion, and why that controls everything
When you run a golden set and report "82 percent passed," you have measured a proportion: the fraction of examples that met the bar. Proportions come with well-understood sampling error. If you re-drew a fresh set of the same size from the same distribution, the observed rate would wobble, and that wobble is your confidence interval.
The key property is that the interval narrows with the square root of sample size, not linearly. Quadrupling the set only halves the interval. That is the entire reason the guidance plateaus around a few hundred examples: you are deep into diminishing returns, where each additional hundred examples shaves off a fraction of a percentage point.
The normal approximation for the half-width of a 95 percent interval on an observed proportion p over n examples is the canonical relation to remember:
At p around 0.8 and n = 300, that half-width is roughly 4.5 points. At n = 100 it is about 7.8 points; at n = 500 about 3.5 points. This is the curve that makes 100 to 500 the practical sweet spot.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI Evals and Promptfoo both report pass rate alongside run to run variance, surfacing whether a score delta is signal or judge noise.
- LangSmith dataset views let teams stratify eval examples by tag, making per-slice example counts visible before sizing decisions.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you actually compute the eval set size needed to detect a 3-point regression at 95 percent confidence?
Treat pass rate as a proportion, use the normal approximation for a difference in proportions, plug in your baseline rate and target minimum detectable effect, then solve for n. McNemar's test fits paired before and after comparisons on the same examples.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking a round number like 1,000 without asking what effect size you must detect or whether every slice has enough examples to give a usable per-slice estimate.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.