What is the primary pitfall of building your eval set exclusively from production traffic logs?
A production-only eval set drifts with the traffic mix and starves rare high-value failure modes, so you lose a stable baseline and undercount the cases that actually hurt.
Imagine grading a chef only on the dishes customers happen to order this week. If the crowd switches from pizza to salads, this week's higher score might just mean salads are easier, not that the chef improved. You also never test the tricky banquet dishes nobody orders day to day, even though those are exactly where a weak chef gets exposed. A good test menu is fixed and deliberate: it keeps the same dishes week over week so scores stay comparable, and it intentionally includes the hard, rare orders. Production logs alone give you a menu that keeps shifting under your feet and skips the scariest dishes, so your grades drift and your blind spots stay invisible.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A golden eval set is the fixed, curated set of inputs you score every model version against. Its job is to be a stable measurement instrument: when the score moves, the model is the only thing that moved. The moment the set itself shifts under you, the number stops meaning what you think it means.
This question targets a specific anti-pattern: building that set exclusively from production traffic logs. It feels natural (use real data) but it quietly breaks the two properties a golden set exists to provide, comparability over time and coverage of the cases that actually matter. The deep dive walks both defects, then the sampling discipline that fixes them.
Why a fixed distribution is the whole point
An eval score is only interpretable relative to a held-constant input distribution. If you freeze 500 inputs and score v1 then v2 on the same 500, a delta is attributable to the model. That is the entire value proposition of a golden set: it isolates the variable you care about. Think of it the way a lab holds every reagent constant except the one being tested. A golden set holds the inputs constant so the model is the only experimental variable.
Production traffic is non-stationary. The intents, phrasings, languages, and entity mix all move week to week, driven by seasonality, product changes, marketing, and external events. A set drawn from logs inherits that motion. A holiday spike, a new onboarding flow, or a viral support thread can reshape the query mix overnight, and none of that has anything to do with model quality.
The failure shows up the first time you reseed the set from recent traffic and the score jumps. You cannot tell whether the model got better or whether the new sample happened to be easier. The measurement and the population moved together, and you can no longer decompose the change.
This matters most for the decisions a golden set is supposed to support: shipping a new model version, gating a prompt change, rolling back a regression. Every one of those decisions assumes the test is constant. The moment the test moves with the traffic, the decisions are made on noise that looks like signal.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith datasets let teams pin a frozen golden split for regression gating separate from a rolling production-sampled split for drift detection.
- RAGAS supports building stratified test sets that deliberately include adversarial and edge-case queries rather than only sampled production logs.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide per-slice quotas for a stratified golden set instead of just copying the traffic frequencies?
Weight by business value and incident cost, not raw frequency. Ensure each slice has enough examples for statistical power on its own metric. Reweight at aggregation if you also need a population-level number.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Drawing the eval set only from production traffic. The set drifts as the traffic mix changes, and rare high-value failure modes stay underrepresented, so score gains become uninterpretable.
60 second bullets to scan on the way to the call.
Why a fixed input distribution is what makes scores comparable
How traffic drift confounds model improvement with query mix changes
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.