Online vs Offline Evaluation
Two evaluation regimes for LLM systems
Offline eval runs against fixed datasets before deployment; online eval measures real users after deployment. You need both, and neither is a substitute for the other.
Offline evaluation
Glossary →Run the system against curated datasets with known answers (or reference completions) and score with metrics, human raters, or an LLM judge. Reproducible and fast.
Best for: Regression tests before ship.
Online evaluation
Glossary →Measure real-user metrics in production: A/B tests, thumbs, engagement, task completion, feedback. The only thing that measures the thing you actually care about.
Best for: Measuring real user value in production.
At a glance
| Dimension | Offline evaluation | Online evaluation |
|---|---|---|
| Where | Pre-deployment | Post-deployment |
| Signal | Fixed datasets + metrics/judge | Real users, A/B, feedback |
| Reproducibility | High | Low |
| Cost | API + annotation | Traffic + risk + infra |
| Fast to iterate | Yes | No (needs statistical power) |
| Best for | Regression testing, model selection | True user impact, discovery |
Key differences
- 1Offline is reproducible; online is inherently noisy
- 2Offline benchmarks proxy for user value; online measures user value
- 3Offline finds regressions before shipping; online detects surprises after
- 4Online eval requires enough traffic + guardrails; offline only needs data
- 5Neither is a substitute: offline can pass while online tanks (and vice versa)
In the interview
- Shipping only because offline benchmarks look good
- Judging a model only from thumbs and no regression tests
- Treating LLM-as-judge as ground truth without human spot-checks
How to choose
Offline for regressions and iteration; online for real user impact. Ship neither without the other.
Common misconceptions
Myth: Offline benchmarks are enough if they're comprehensive.
Reality: Static datasets go stale, users find edge cases you didn't test, and gaming benchmarks is Goodhart's law in action.
Myth: Online eval alone is enough for a mature product.
Reality: You still need offline gates or a small change silently breaks a rare but critical case.
Memory aid
Offline: exam under lab conditions. Online: how the surgeon actually performs. Both count.
Can you combine them?
Absolutely. The mature loop is: offline gate before deploy, canary online with metrics, roll out on success. Findings from online failures become new offline test cases.