What is the primary pitfall of building your eval set exclusively from production traffic logs?

A **golden eval set** is a fixed, curated collection of inputs you score your model against every time you change it. The whole point is comparability: if the set stays fixed, a higher score means a better model. Building that set only from production logs breaks this in two ways. First, **distribution shift**: the mix of queries users send changes month to month. If you keep refreshing the set from recent traffic, a score jump might just mean this month's queries were easier, not that your model improved. You lose the stable baseline. Second, rare-case coverage: production traffic is mostly normal usage. The dangerous failures (edge cases, adversarial inputs, low-frequency query types) are exactly the ones users rarely send, so they barely appear in the logs. Your eval ends up measuring the easy middle and ignoring the scary tails. The other options are wrong. Memory is a non-issue, you sample anyway. PII is a real concern but is solved by scrubbing, not a hard legal ban. And production queries are not uniformly easy, real users send plenty of messy, malformed inputs. The fix is to blend production samples with deliberately constructed hard cases and freeze the set so scores stay comparable.

A **golden eval set** must be stable and representative. Drawing it only from production traffic violates both. The first failure is drift. The traffic distribution shifts over time. If the eval set is just recent logs, a score change confounds two things. Did the model improve, or did the query mix get easier? You lose a fixed baseline, so version to version comparisons stop being trustworthy. The second failure is rare-case starvation. Production logs reflect normal usage. High-value failure modes are low-frequency by definition: adversarial prompts, edge cases, safety-sensitive queries. Uniform sampling from logs underweights exactly the cases that hurt most in production. The fix is stratified, deliberate construction. Bucket by slice, difficulty, and query type. Sample to a target distribution rather than the raw traffic distribution. Oversample rare high-value slices so the eval has statistical power on them. Then freeze the set so it stays a stable baseline. The other options are distractors. Memory is not the constraint, you sample regardless. PII is solved by scrubbing, not a legal ban. And real traffic is not uniformly easy, users send plenty of malformed inputs. The genuine pitfalls are drift and rare-case coverage.