Offline eval now or wait for the A/B test results? Pick the key difference between online and offline evaluation.
Offline eval uses a fixed dataset before deployment for speed and reproducibility; online eval uses live traffic after deployment for realism but is slower and riskier.
Imagine you are testing a new recipe. Offline evaluation is cooking the dish in your kitchen with your own ingredients, tasting it yourself, and tweaking it before serving anyone. You can repeat the same test as many times as you like with the same ingredients. Online evaluation is serving the dish to actual diners and watching their reactions. You learn what real people think, but if the dish is bad, real customers had a bad experience. You test offline first so you catch obvious problems cheaply. You test online second so you learn things your kitchen test could never reveal, like whether diners actually order the dish when they see the menu.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: define both modes, name the key tradeoff (reproducibility vs realism), explain the two-stage gate pattern, give one example of each, and close with why neither alone is sufficient.
Real products, models, and research that use this idea.
- Chatbot Arena is an online evaluation system: real users submit prompts and vote on live, blind model comparisons, capturing preferences a static benchmark cannot.
- Most LLM teams at companies like Anthropic and Google run offline eval suites in CI that gate every prompt change before it reaches production traffic.
- Shadow deploys at large search companies run a new ranking model on live queries without surfacing results, measuring quality offline on real inputs.
- A/B tests on customer-facing chatbots typically require thousands of conversations to reach statistical power, making online evaluation a multi-day commitment.
- Promptfoo and Braintrust both support offline evaluation as a CI step, then recommend online monitoring dashboards to catch post-deploy drift.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide when an offline dataset is stale and needs refreshing?
QWhat is a shadow deploy and when would you use it instead of an A/B test?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming offline eval alone is sufficient. Offline datasets cannot capture distribution shift, user behavior diversity, or production edge cases that only appear under real traffic.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.