You are setting up the golden dataset that will run on every prompt PR for a B2B support assistant. Describe how you choose examples, how you label them, how you keep it fresh, and how you avoid the trap where the model memorizes the set.
Stratify 100 to 300 production-derived examples across tenant, intent, difficulty, and outcome; capture context snapshots and tolerance bands; append every escaped incident; rotate a held-out slice to defend against
Think of a driving test for a self-driving car. Random street footage will not catch the rare hard cases (kids chasing balls, snowed-over signs). So you build a deliberate test pack: city, highway, night, rain, school zone, one each of the rare scary stuff. You write down what 'passing' looks like for each scene (slow down, change lane, stop). You record the camera feed, not just the steering wheel angle, because you want to know what the car saw. Every real incident on the road becomes a new test in the pack. And you keep some scenes secret so the engineers cannot just train on the test. A golden set for a chatbot works the same way: deliberate coverage, snapshots of what the system saw, tolerance for multiple valid behaviors, and a defense against the team accidentally teaching to the test.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A golden set is the gate that decides whether a prompt PR ships. The design of the set determines whether the gate actually catches the regressions that matter. A poorly designed set passes regressions that hurt the enterprise tier while celebrating wins on the long tail of easy questions. A well-designed set catches the rare but expensive failures before they reach a customer.
This explanation walks through the four design pillars: sourcing and stratification, per-example contract, labeling and freshness, and contamination defense. Each is a place where the casual approach fails and the production approach pays off.
Sourcing and stratification
The seed for any production golden set is real production traffic, not synthetic prompts written by the team. Synthetic prompts encode what the team thinks the assistant has to handle. Real traces encode what users actually ask, which is always broader, weirder, and more skewed than the team imagines.
The trap is to sample randomly. Random sampling reproduces the natural distribution, which is dominated by easy cases. Forty percent of B2B support traffic is some flavor of 'where do I click', and a golden set that mirrors that ratio will have its score moved almost entirely by performance on click-location questions. The hard escalated cases, which are 5 to 10 percent of natural traffic but 80 percent of contract-impact regressions, are barely represented.
The fix is deliberate stratification across the dimensions that matter: tenant tier (free, paid, enterprise, strategic), intent category (refunds, billing, onboarding, technical, abuse, churn-risk), difficulty (single-turn lookup, multi-turn clarification, multi-step reasoning), and outcome (resolved-positive, escalated-human, abandoned, complained). The hard slices get oversampled relative to natural distribution: a 100 to 300 example set might allocate 40 to 50 percent of its budget to escalated and complained cases even though those are a small fraction of total traffic.
This is the same logic that test engineers in safety-critical domains have used for decades. The rare cases are where the bugs hide; the test pack has to over-represent them.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's customer support stack: golden set stratified by abuse-risk intent so jailbreak attempts gate as their own slice.
- Stripe's developer-support assistant: every escalated incident from the last quarter is appended to the next golden set release.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you handle the case where the policy itself changes and yesterday's correct answer becomes today's wrong answer?
Version the policy as content-addressed bytes and attach the policy-version reference to each golden example. When the policy changes, the affected examples are flagged for relabel; the old version is retained for historical comparisons but removed from the active gate. The gate must point at the active policy version, not a hardcoded answer string.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Building the golden set by random-sampling production traffic. Random sampling oversamples the easy cases and undersamples the rare hard ones that actually move quality, so the gate passes regressions that matter to a small but important slice.
60 second bullets to scan on the way to the call.
What stratification dimensions matter for a B2B support workload
What gets stored per example beyond question and answer
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.