Spot the flaw in this eval set construction approach

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Two flaws: uniform sampling buries rare failure modes, and monthly refresh breaks baseline comparability. Fix with stratified sampling plus a frozen baseline slice separate from a refresh slice.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading a school by testing 500 random students each month. Two things go wrong. First, you pick students at random, so the few kids who are struggling badly almost never show up in your sample, yet those are exactly the cases you most need to catch. Second, you swap in a brand-new set of students every month, so when this month's average is higher than last month's, you cannot tell whether the school actually improved or you just happened to test easier kids. The fix: deliberately include some of every kind of student, not just the common ones, and keep one fixed group of the same students every month so month to month numbers actually mean something.

Key concepts

Golden set design Stratified sampling Baseline comparability Distribution shift

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: two flaws (uniform sampling kills coverage, monthly refresh kills comparability) plus the stratified and frozen-baseline fix, leakage and drift concerns, and how to report per slice.

Real-world examples

Real products, models, and research that use this idea.

LangSmith and Braintrust default to version-pinned datasets with per-slice scoring so a frozen baseline stays comparable across model releases.
RAGAS users build stratified eval sets by query type and seed adversarial cases rather than relying on uniform production draws.
OpenAI Evals and Promptfoo support tagging examples by category so teams can report per stratum rather than one blended mean.
Anthropic and OpenAI eval teams keep frozen regression suites separate from rolling drift-detection sets to decouple model effect from data churn.
Patronus and Galileo surface near-duplicate and train-eval leakage detection so production-sourced golden sets do not inflate scores.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you size and weight strata so rare failure modes are visible without distorting the headline score?

Sign in or go Pro to see the approach for this follow-up.

QHow do you detect that your frozen baseline slice has gone stale relative to production?

Sign in or go Pro to see the approach for this follow-up.

QWhat contamination risks come from sourcing a golden set straight from production logs?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating a uniform random draw from production as a complete eval set. Rare failure modes vanish, and refreshing the set monthly makes month over month scores incomparable.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.