Contrast offline (golden set) and online (sampled traffic) evaluation as complementary signals
Offline evals gate PRs on a fixed golden set; online evals catch novel failures and distribution drift on real traffic. Each covers what the other misses.
Picture a restaurant kitchen. Offline evals are like a chef tasting the same five signature dishes every morning before the doors open, scoring each one. If today's batch fails the taste test, the kitchen does not open. Online evals are like a manager walking the dining room every hour, listening to what real customers actually ordered and how they reacted. The morning taste test catches recipe drift. The dining-room walk catches the new dish a customer requested that nobody planned for. You need both. Skipping the morning taste lets a bad recipe out the door. Skipping the dining-room walk means you only find out about new problems when a complaint reaches the email inbox.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Offline and online evaluation often get talked about as competing approaches, as if a team should pick one and call it a strategy. They are not competing. They cover different halves of the failure space, and a mature LLM product runs both because skipping either leaves a real class of failures uncovered.
Offline evals are the part of evaluation that lifted directly out of classical ML: a curated dataset with known expected behaviors, a deterministic scoring function, a regression test that runs on every code change. Online evals are the new piece LLMs forced: a sampled stream of production traffic scored by an LLM-as-judge, surfaced as dashboards, alerting when the score trends down.
This walkthrough covers what each is designed to catch, the specific failure modes each one misses, the cost and bias considerations that come up at production scale, and the operational rules that govern how they interact in a mature stack.
One-line summary: offline is the gate that decides whether a PR ships; online is the watchtower that catches everything that ships and then breaks for reasons the gate could not see.
What offline evals are designed to catch
The shape
A golden set is a fixed list of input-output pairs (or input-rubric pairs for graded scoring) curated by hand. Every PR that touches the prompt runs the entire set; each response is scored against the rubric; the aggregate score is compared to the baseline; significant drops block merge.
What this catches
Regressions on cases you have already identified as important. The exact phrasing that broke last quarter and prompted the rule change. The five edge cases the PM listed as launch criteria. The adversarial inputs the safety team added after the incident review.
What this misses
Anything not in the set. Novel inputs from users in a new geography. Inputs that involve a new product feature shipped after the set was last updated. Subtle distribution shifts in input length, language mix, or topic. All of these can break a prompt in production while every offline eval continues to pass.
Why the set decays
Golden sets are launch deliverables for most teams; nobody owns ongoing maintenance. Six months in, the set covers 50 percent of the failure modes the system actually exhibits in production. A year in, maybe 30 percent. The gate keeps reporting green while the watchtower (if you have one) is on fire.
Defense
Treat the set as a living artifact. Every incident produces a case that goes into the set. Every postmortem adds at least one entry. Schedule monthly reviews that pull novel failures from the online stream into the offline set.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepEval and Promptfoo are the standard offline harness for golden-set regression gates in CI; both integrate with GitHub Actions for PR comments.
- LangSmith and Langfuse both ship online-eval samplers that score production traces with an LLM-as-judge and surface time-series dashboards.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you maintain the offline golden set so it does not freeze at launch?
Every incident produces a failure case; every postmortem adds that case to the set. Schedule a monthly review where on-call surface novel failure modes from the online stream into the set. The set is a living artifact, not a launch deliverable.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating offline pass as production-ready. The golden set covers what you knew to test for; new failure modes always appear first in live traffic.
60 second bullets to scan on the way to the call.
The CI-gate role of offline evals on a fixed golden set
The watchtower role of online evals on sampled production traffic
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.