When you build a new prompt or swap a model, you need to know whether the change actually makes things better. There are two broad ways to find out, and they run at different stages.
**Offline evaluation** happens before deployment. You assemble a fixed dataset of inputs and expected outputs, run the new prompt against it, and score the results. Because the dataset is fixed, you can repeat the same test tomorrow and get the same result. It is fast, cheap, and fully reproducible.
Online evaluation happens after deployment. You send some fraction of live traffic to the new version and compare real user outcomes, clicks, thumbs-up, task completion, against the old version. This is an A/B test, a shadow deploy, or a canary rollout. It captures things offline evaluation cannot: how users actually phrase questions, how traffic patterns shift over time, and whether the new version breaks under real load.
The key difference is the input source. Offline controls inputs; online reflects reality. Good teams run both. Offline catches regressions cheaply before anyone is affected. Online confirms the change works under conditions the dataset could never fully represent.
The key difference sits in when each mode runs and what inputs it uses.
**Offline evaluation** runs before deployment against a curated, frozen dataset. You control every input, so results are reproducible and fast to iterate on. A developer can run the same suite fifty times in an afternoon while tuning a prompt. The limitation is that a fixed dataset cannot represent every real user query, and if the live distribution shifts, the offline score becomes stale.
Online evaluation runs after deployment against live user traffic. A/B tests, shadow deploys, and canary rollouts all fall here. The signal is grounded in reality: real queries, real latency, real user satisfaction. The cost is risk (real users see regressions), speed (you need statistical power, which means waiting for enough traffic), and expense (production infrastructure, monitoring, incident response).
The practical pattern in 2026 teams: offline evaluation gates every change before it ships. Online evaluation validates the change under real conditions once it passes the offline bar. Neither replaces the other. Skipping offline means shipping untested changes. Skipping online means trusting that a dataset fully represents production, which it never does.
Quick reference
TL;DR
Offline eval uses a fixed dataset before deployment for speed and reproducibility; online eval uses live traffic after deployment for realism but is slower and riskier.
Imagine you are testing a new recipe. Offline evaluation is cooking the dish in your kitchen with your own ingredients, tasting it yourself, and tweaking it before serving anyone. You can repeat the same test as many times as you like with the same ingredients. Online evaluation is serving the dish to actual diners and watching their reactions. You learn what real people think, but if the dish is bad, real customers had a bad experience. You test offline first so you catch obvious problems cheaply. You test online second so you learn things your kitchen test could never reveal, like whether diners actually order the dish when they see the menu.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Internal tool with no live users yet There is no live traffic to run online evaluation against. Offline evaluation is the only option until you have users.
High-stakes regulated domain where any regression could cause harm Online evaluation exposes real users to the new version. In regulated settings, extensive offline evaluation and staged rollouts with human oversight may be required before any online exposure.
One-off exploratory prompt experiment Setting up a full A/B test infrastructure is overkill. A quick offline eval on a handful of examples is sufficient to decide whether to pursue the idea further.
Quick scan · 2–4 min
Everything important, quickly.
5 min: define both modes, name the key tradeoff (reproducibility vs realism), explain the two-stage gate pattern, give one example of each, and close with why neither alone is sufficient.
Real-world examples
Real products, models, and research that use this idea.
1Chatbot Arena is an online evaluation system: real users submit prompts and vote on live, blind model comparisons, capturing preferences a static benchmark cannot.
2Most LLM teams at companies like Anthropic and Google run offline eval suites in CI that gate every prompt change before it reaches production traffic.
3Shadow deploys at large search companies run a new ranking model on live queries without surfacing results, measuring quality offline on real inputs.
4A/B tests on customer-facing chatbots typically require thousands of conversations to reach statistical power, making online evaluation a multi-day commitment.
5Promptfoo and Braintrust both support offline evaluation as a CI step, then recommend online monitoring dashboards to catch post-deploy drift.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide when an offline dataset is stale and needs refreshing?
Sign in or go Pro to see the approach for this follow-up.
QWhat is a shadow deploy and when would you use it instead of an A/B test?
Sign in or go Pro to see the approach for this follow-up.
QHow do you handle the statistical power problem in online A/B tests for LLM features?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming offline eval alone is sufficient. Offline datasets cannot capture distribution shift, user behavior diversity, or production edge cases that only appear under real traffic.