What is the practical guidance on golden eval set size for LLM systems?

A **golden eval set** is a fixed, curated set of examples you run your LLM system against to track quality over time. The classic guidance is that 100 to 500 well-chosen examples usually suffice for most use cases. The reason is statistical. An eval score like pass rate is just a proportion, and a few hundred examples already pin that proportion down to within a few percentage points. Doubling from 500 to 1,000 only shrinks that uncertainty a little, so the extra labelling effort buys you very little. What matters far more than raw count is two things. First, coverage: does the set include your known failure modes and every important slice of traffic? Second, variance: if you run the same set twice and the score moves, your measurement is noisy. The other options are wrong. You do not need 10,000 examples for a score to mean something. Averaging more runs does not fix a set that misses your failure modes. And eval size has nothing to do with model parameter count.

Golden set size is driven by two things: the **effect size** you must detect, and per-slice coverage. It is not a round number you pick in advance. The statistical floor comes from the fact that a pass rate is a proportion. A confidence interval on a proportion narrows with the square root of sample size. So a few hundred examples already gives a half-width of a few points. Going from 500 to 5,000 shrinks the interval only by about 3x, for 10x the labelling cost. The real driver is the smallest quality change you care about. To reliably detect a 2-point regression, you need many more examples than to detect a 10-point one. That math sets your floor. Coverage then multiplies it. If you report metrics per slice (query type, language, customer tier), each slice needs enough examples on its own. Five slices needing 100 each means 500 total, even when the headline number stabilises sooner. The 100-to-500 guidance is the typical answer once you combine a moderate effect size with a handful of slices. The other options confuse size with significance. Variance, not raw count, is usually the binding constraint: pin the judge to temperature 0 before adding examples.