Offline eval uses a fixed dataset before deployment for speed and reproducibility; online eval uses live traffic after deployment for realism but is slower and riskier.
Imagine you are testing a new recipe. Offline evaluation is cooking the dish in your kitchen with your own ingredients, tasting it yourself, and tweaking it before serving anyone. You can repeat the same test as many times as you like with the same ingredients. Online evaluation is serving the dish to actual diners and watching their reactions. You learn what real people think, but if the dish is bad, real customers had a bad experience. You test offline first so you catch obvious problems cheaply. You test online second so you learn things your kitchen test could never reveal, like whether diners actually order the dish when they see the menu.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every LLM team eventually faces a fork in the road: do you evaluate a change against a test set on your laptop, or do you ship it and watch what happens with real users? The answer, always, is both. But understanding why requires unpacking what each mode can and cannot see.
This deep dive covers the mechanics of offline and online evaluation, the tradeoffs that make them complementary, and the standard two-stage pattern that production teams follow in 2026.
Offline evaluation: the controlled laboratory
Offline evaluation runs before deployment. You assemble a dataset of input prompts and, optionally, reference outputs or rubrics. You run the model or prompt variant against every input, score the results with automated metrics or an LLM judge, and get a report.
The defining property is that inputs are fixed and controlled. You can run the same suite on Monday and Friday and compare results directly. This makes offline evaluation deterministic, fast (seconds to minutes for a few hundred examples), and cheap (no production infrastructure required). It is the natural fit for CI pipelines: every pull request triggers the eval suite, and regressions block the merge.
The weakness is coverage. A curated dataset is a sample of the world, not the world itself. If user behavior shifts, if new query types emerge, or if production load introduces latency-dependent failure modes, the offline score becomes a stale proxy. The dataset captures the distribution at the time it was built, and that distribution drifts.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Chatbot Arena is an online evaluation system: real users submit prompts and vote on live, blind model comparisons, capturing preferences a static benchmark cannot.
- Most LLM teams at companies like Anthropic and Google run offline eval suites in CI that gate every prompt change before it reaches production traffic.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide when an offline dataset is stale and needs refreshing?
Track the distribution of live queries (topic, length, intent) and compare against your eval set. When the overlap drops below a threshold, sample fresh examples from production logs and re-curate.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming offline eval alone is sufficient. Offline datasets cannot capture distribution shift, user behavior diversity, or production edge cases that only appear under real traffic.
60 second bullets to scan on the way to the call.
Define offline evaluation as pre-deployment against a fixed dataset
Define online evaluation as post-deployment against live traffic
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.