How large should a golden eval set be, and what signals tell you when you have enough examples?

A **golden eval set** is a curated, stable list of examples you run a model against to measure quality. The instinct is to ask 'how many?' but that is the wrong question. The right one is: what decision will this set make, and how small a difference must it detect? A common baseline is 100 to 500 well-chosen examples, but that number is downstream of the job. To catch a big 10-point drop, a small set works. To catch a 2-point regression, you need many more examples to see that signal through the noise. You have enough when two things hold. First, running the same set twice gives nearly the same score, so the run to run wobble is smaller than the difference you care about. Second, every known failure category and important query type has its own examples, not just the overall total. You need more when a slice, say refund questions, has only a handful of examples, when you find a new failure mode the set never covered, or when scores jump around between identical runs. The headline mistake is fixating on total count. A stratified set of 300 beats a random pile of 3,000 that misses your hard slices.

Size the set to the decision, not to a folklore number. The real question is the smallest score difference you must reliably detect, the **minimum detectable effect**. Start from the confidence interval on a pass rate. For a proportion the half-width is: $$\text{CI}_{95} \approx 1.96 \sqrt{\frac{p(1-p)}{n}}$$ A worst-case rate of 0.5 at n=100 gives a half-width near 10 points. At n=400 it halves to 5. Resolution improves only as the square root of n, so doubling precision costs four times the examples. Two stopping signals say you have enough. Run to run variance on repeated passes is small relative to your regression threshold. And every failure category and query slice clears its own minimum, roughly 30 examples for a stable rate. Three signals say you need more. A slice has too few examples to score. You discover a failure mode the set never covered. Or repeated runs wobble more than the difference you act on. The trap is per-slice resolution. A total of 1,000 examples can still leave a critical slice with only 8. That slice has a confidence interval near 35 points wide. So each slice needs its own n, which is why stratified sizing beats raw count almost every time.