Design eval infrastructure that makes a weekly model swap boring
Leadership wants to run on whichever model is cheapest while clearing the quality bar, re-decided every week as providers ship updates and price cuts. Design the eval infrastructure that turns a weekly model swap into a routine, low drama decision instead of a fire drill.
Versioned golden sets, an automated harness with per-model prompt variants, a calibrated judge, threshold-based gates, and a shadow stage with one-click rollback turn the swap into a routine decision.
Imagine choosing the best brand of soap every week. You do not sniff each one and argue. You set up the same test every time: how well does it clean, how much does it cost, does it irritate skin. Then you score each brand on those tests automatically. If a new brand passes the bar, you switch. If it does not, you keep the old one. The decision is boring because the criteria are written down in advance. The trick is that the testing kit itself has to be trustworthy. Your sniff-test panel can drift over time; the recipes can be tuned to favor the brand you already use. Eval infrastructure for LLM swaps is the same idea, with a few extra traps around prompt portability and judge drift.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A weekly model swap is a forcing function. Most teams that talk about model agility do not actually swap weekly, because every swap is a debate. The debate happens because the team is making qualitative judgments on partial information, and qualitative judgments need a quorum. Eval infrastructure removes the debate by replacing it with thresholds.
The design has five layers. Data: versioned golden sets that track real traffic. Harness: identical automated runs per candidate. Judge: an LLM scorer calibrated against humans. Decision: threshold-based gates and a shadow stage. Operations: pinned model IDs, one-click rollback, and contamination hygiene.
This deep dive walks each layer, names the failure mode that hits it, and shows how the five layers interlock so the weekly meeting can be ten minutes long.
Golden datasets that track real traffic
A golden set has to look like the traffic it gates. Hand-curated sets drift apart from production within weeks because product changes, user behavior changes, and content sources change faster than humans curate. The robust pattern samples from production logs on a schedule, grades the samples via human plus synthetic labeling, and freezes a versioned snapshot.
Each use case gets its own set. A single bundle of one thousand mixed examples masks regressions on the long tail. Splitting by use case (summarization, structured extraction, agentic tool use, retrieval-grounded answering) keeps the score readable and lets specific gates apply to specific surfaces.
The set is held out from prompt tuning. The moment the same examples drive prompt iteration, the test is no longer a test. Maintain a separate development set for tuning, and the golden set strictly for evaluation. Refresh the golden set on a slower schedule (monthly is typical) so weekly comparability is preserved while the set still tracks reality.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Braintrust, Langfuse, and Helicone offer hosted eval pipelines with versioned datasets and threshold-based gates
- OpenAI Evals and Anthropic's evaluation tooling support reproducible suites that work across Claude Opus 4.7, Sonnet 4.6, and GPT-5.5
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect judge drift?
Maintain a small fixed gold set with frozen human labels. Run the judge against it every week and track agreement rate. When agreement drops past a threshold, re-anchor the judge prompt or rotate to a fresh judge model and re-calibrate.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Running the new model with the old prompt and concluding the new model is worse, when the real finding is that prompts do not port across models without a per-model variant.
60 second bullets to scan on the way to the call.
What makes a golden set production-sampled, versioned, and held out
Why each candidate needs its own prompt variant and how to version those
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.