Spot the flaw in this eval set construction approach

A **golden set** is the fixed collection of examples you score every model version against. This setup makes two mistakes that quietly corrupt that signal. The first mistake is uniform random sampling from production traffic. Most production queries are easy and common. The cases that actually break your model (adversarial inputs, edge cases, rare query types) show up far less often, so a uniform draw of 500 barely includes any of them. Your score looks great while the failures that matter stay invisible. The fix is stratified sampling: group queries by type, then sample enough from each group, including a deliberate slice of hard and adversarial cases. The second mistake is refreshing the whole set every month. If the examples change between benchmarks, a higher score next month might mean the model improved, or it might mean this month's examples were easier. You cannot tell. The fix is to keep a frozen baseline slice that never changes, and add a separate refresh slice for tracking new behavior. The frozen part gives you a comparable number over time; the refresh part keeps you current.

The setup has two distinct flaws. First flaw: **uniform sampling** from production underrepresents rare failure modes. Production traffic is dominated by easy, common queries. A uniform draw of 500 systematically misses the low-frequency cases that break models. The fix is stratified sampling by query type. Add a deliberate adversarial slice that you seed and perturb rather than wait for in traffic. Second flaw: monthly refresh of the whole set destroys baseline comparability. If the examples change between benchmarks, a score delta is confounded. You cannot tell a real model improvement from an easier example mix. The fix is a frozen baseline slice that never changes, scored every release for true regression detection. Keep both concerns separate. Use a frozen baseline slice for comparable trend lines across model versions. Use a separate, periodically refreshed slice to track distribution shift in real user behavior. Reporting one blended number to stakeholders hides which signal moved. The deeper point: an eval set is an instrument. Changing the instrument and the thing it measures at the same time gives you noise, not insight.