A **golden eval set** is a fixed, curated set of examples you run your LLM system against to track quality over time. The classic guidance is that 100 to 500 well-chosen examples usually suffice for most use cases.
The reason is statistical. An eval score like pass rate is just a proportion, and a few hundred examples already pin that proportion down to within a few percentage points. Doubling from 500 to 1,000 only shrinks that uncertainty a little, so the extra labelling effort buys you very little.
What matters far more than raw count is two things. First, coverage: does the set include your known failure modes and every important slice of traffic? Second, variance: if you run the same set twice and the score moves, your measurement is noisy.
The other options are wrong. You do not need 10,000 examples for a score to mean something. Averaging more runs does not fix a set that misses your failure modes. And eval size has nothing to do with model parameter count.
Golden set size is driven by two things: the **effect size** you must detect, and per-slice coverage. It is not a round number you pick in advance.
The statistical floor comes from the fact that a pass rate is a proportion. A confidence interval on a proportion narrows with the square root of sample size. So a few hundred examples already gives a half-width of a few points. Going from 500 to 5,000 shrinks the interval only by about 3x, for 10x the labelling cost.
The real driver is the smallest quality change you care about. To reliably detect a 2-point regression, you need many more examples than to detect a 10-point one. That math sets your floor.
Coverage then multiplies it. If you report metrics per slice (query type, language, customer tier), each slice needs enough examples on its own. Five slices needing 100 each means 500 total, even when the headline number stabilises sooner.
The 100-to-500 guidance is the typical answer once you combine a moderate effect size with a handful of slices. The other options confuse size with significance. Variance, not raw count, is usually the binding constraint: pin the judge to temperature 0 before adding examples.
Quick reference
TL;DR
Golden set size is driven by the effect size you must detect and per-slice coverage, not a round number. 100 to 500 well-stratified examples usually suffice; the pass-rate confidence interval tells you if you need more.
Imagine polling whether a town likes a new park. You do not poll everyone. A few hundred well-chosen people give a tight enough estimate. But if you want to know what each neighborhood thinks separately, each neighborhood needs its own sample, so the total grows. An eval set works the same way. A few hundred examples pins down your overall pass rate to within a few points. The catch: if you care about each customer type or each failure mode separately, each slice needs enough examples on its own. And if you re-run the same set and the score jumps around, adding examples will not save you. You first have to make the measurement stable, by pinning the judge to a fixed temperature.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very rare but high-severity failure modes such as safety violations or data leakage A proportion-based size gives almost no rare examples. Oversample or build a dedicated targeted suite instead.
Tasks with cheap exact-match ground truth at huge scale, like classification or extraction Labelling is nearly free, so a much larger set is fine and tightens the estimate at little cost.
Reporting many fine-grained slices simultaneously in one dashboard Each slice needs its own minimum, so a single global size target badly under-samples the small slices.
Measurement is dominated by a stochastic judge at default temperature Noise, not sample size, binds. Fix the judge to temperature 0 before any sizing decision matters.
Quick scan · 2–4 min
Everything important, quickly.
4 min: the proportion CI sets the floor, effect size drives the number, slices multiply it, and judge variance usually binds before raw count does.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI Evals and Promptfoo both report pass rate alongside run to run variance, surfacing whether a score delta is signal or judge noise.
2LangSmith dataset views let teams stratify eval examples by tag, making per-slice example counts visible before sizing decisions.
3RAGAS faithfulness scoring is typically run over a few hundred stratified RAG examples rather than tens of thousands.
4Anthropic and OpenAI eval guidance emphasises curated failure-mode coverage over raw example count for golden sets.
5Braintrust and Langfuse expose confidence intervals on aggregate scores so teams can judge whether a regression is statistically real.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you actually compute the eval set size needed to detect a 3-point regression at 95 percent confidence?
Sign in or go Pro to see the approach for this follow-up.
QYour global eval score is stable but one customer slice keeps swinging. What is happening and what do you do?
Sign in or go Pro to see the approach for this follow-up.
QWhen does adding more examples genuinely help versus when is variance the real bottleneck?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Picking a round number like 1,000 without asking what effect size you must detect or whether every slice has enough examples to give a usable per-slice estimate.