Online evals judge a sample of live traffic to catch drift; offline evals replay a fixed dataset to gate deploys. You need both because neither alone covers pre-deploy regression and real-world distribution.
Imagine a restaurant. Offline evals are the standardized recipe tasting you do in the test kitchen before changing the menu: the same dishes, the same testers, every time, so you can fairly compare the old recipe to the new one. Online evals are sending a food critic to a random table every night to score whatever real customers ordered. The kitchen tasting catches recipes you broke before they reach diners. The random table catches the dishes you never thought to test, the ingredient supplier who quietly changed, and the slow drift no one noticed. Skip the kitchen and you ship broken recipes. Skip the random table and you never learn what reality looks like.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM evaluation has settled into two layers that work together. Offline dataset evals are the pre-deploy gate. Online sampled evals are the post-deploy radar. The two answer different questions and catch different failures, and any production stack that takes quality seriously runs both.
This deep dive walks through what each layer is, what each one catches that the other cannot, how to size them in practice (sample rate, dataset size, judge cost), and where each one breaks. By the end you should be able to explain in an interview why a team that ships with online-only or offline-only has a blind spot, and what specific failure that blind spot will eventually produce.
The mental model: offline gates the change you are about to ship; online watches the world the change is shipped into. Either layer alone leaves the other side unguarded.
Offline dataset evals: the pre-deploy gate
An offline eval is a reproducible benchmark. You curate a dataset of inputs (200 to 2,000 examples is the typical range), define expected outputs or rubrics, and re-run the eval on every change to your stack.
What it gives you
- Reproducibility. Same inputs, same scoring function, comparable scores across runs.
- Statistical comparability. Because the input distribution is fixed, paired tests (paired t-test, Wilcoxon) work directly. You can say 'this prompt regresses faithfulness by 4 points, p = 0.002' with confidence.
- Cheap iteration. Cents per run on a 500-example set even with a flagship judge.
- Deploy gating. Wire it into CI so a regression blocks merge.
What it does not give you
- Real distribution. The dataset reflects what you thought to test. If users do something you did not anticipate, the eval is silent.
- Drift detection. The dataset is fixed, so you cannot see the world changing around it.
- Edge cases that emerged post-launch. A new user persona, a new locale, a new product feature, all invisible.
Practical sizing
200 examples is the floor where pairwise tests start having statistical power on a 3-point difference. 2,000 is the ceiling before iteration cost becomes annoying. Most production teams sit around 500. Refresh quarterly by sampling production into the dataset with redaction.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse pairs dataset runs (offline) with online scoring on a sampled subset of traces, both surfaced on the same eval dashboard.
- LangSmith treats datasets as a first-class object and lets online evaluators emit scores back onto live runs via the Annotation Queues API.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide the online sampling rate?
Set a judge-cost budget as a percentage of production cost (typical 1 to 3 percent). Solve sample rate = budget / (judge_cost_per_call / production_cost_per_call). Floor at a rate that gives statistical power on your alert window.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking one eval style and assuming it covers everything. Offline-only ships drift bugs; online-only ships pre-deploy regressions. The two are complementary, not interchangeable.
60 second bullets to scan on the way to the call.
What an offline dataset eval is and what makes it reproducible
What an online sampled eval is and what fraction of traffic typically gets scored
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.