Production traffic has no ground truth labels, yet you need to detect when answer quality drifts. Describe an online eval approach and its limitations.
Sample live traffic, score it with an LLM-as-judge plus implicit user signals, and alarm on drift versus a baseline — knowing both proxies are noisy, not ground truth.
Imagine running a restaurant with no formal food critic. You still want to know if the kitchen is slipping. So you watch two things: a manager who tastes a few random plates each night and rates them, and the customers themselves — are they sending food back, asking for a redo, or leaving early? Neither is a perfect critic. The manager has their own quirks, and silent customers tell you little. But if both signals start sliding at once, something's wrong, and you investigate. Online eval works the same way: an LLM rates a sample of answers, user behavior fills in the rest, and you watch for the trend dropping.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Offline evals are the easy part of LLM quality. You curate a set of cases with known-good answers and grade against them. The hard, senior part is what happens after you ship, when the traffic is real, the volume is large, and absolutely none of it comes with a correct answer attached. A user asked something, the model answered, and nobody on earth has labeled whether that answer was good. Yet a prompt tweak or a silent provider update can degrade quality at any moment, and you need to notice before your users churn.
That's the bind this question puts you in: detect quality drift with no ground truth. It's a favorite for senior LLM-eng rounds because the naive answer — "track thumbs-up rate" — is incomplete in instructive ways, and the strong answer requires you to be honest about how unreliable every available signal is.
This deep dive builds the online-eval system from the no-labels constraint: why you sample, what an LLM-as-judge buys you and what it doesn't, how implicit user signals fill the gaps and where they mislead, and why the whole apparatus is a complement to offline eval rather than a substitute. The recurring theme is calibrated distrust of your own metrics.
Reframing the goal: drift detection, not measurement
The first senior move is to correct what the question seems to ask. Without labels you cannot measure absolute quality in production — there's no oracle to compare against, so 'our accuracy is 91%' is a number you simply cannot compute on live traffic. If your design implies you can, you've misunderstood the constraint.
What you can do is detect change. You establish what 'normal' looks like during a period you trust, then watch for the proxies to move away from it. The output of online eval isn't a grade; it's an alarm that says 'something shifted, go look.' That reframing matters because it sets the bar correctly for your proxies: they don't need to be accurate in an absolute sense, they need to be stable and sensitive — consistent enough that a real regression stands out against their noise floor.
This also explains why you alarm on drift relative to a baseline rather than against a fixed threshold. A judge score of 0.78 means little on its own — is that good? But a judge score that sat at 0.78 for weeks and dropped to 0.64 the day after a prompt change is a loud, actionable signal. You're using the proxy as its own control. The same logic applies to every implicit signal: the absolute regeneration rate is hard to interpret, but a sudden jump in it after a release is not. Design the whole system to surface deltas, and you sidestep the impossibility of absolute measurement.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Offline eval | Online eval |
|---|---|---|
| Labels | Curated ground truth | None — proxies only |
| Signal | Exact-match / rubric on fixed set | LLM-judge + implicit user signals |
| Strength | Trustworthy, repeatable gate | Real traffic, catches live drift |
| Weakness | Can miss real-world cases | Noisy, biased, not ground truth |
| Role | Pre-release CI gate | Drift smoke alarm in production |
Real products, models, and research that use this idea.
- LangSmith / Langfuse online evaluators scoring a sampled stream of production traces with an LLM-as-judge.
- Chat products using thumbs up/down, regeneration clicks, and message edits as implicit quality signals.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you stop an LLM-as-judge from sharing the generator's blind spots?
Discuss using a different model family or stronger judge, reference-grounded grading, ensembling judges, and calibrating against human labels to bound bias.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Trusting an LLM-as-judge score as if it were ground truth, never calibrating it against human labels or noticing it shares the model's own blind spots.
60 second bullets to scan on the way to the call.
Why production traffic can't be scored with exact-match accuracy
How to use an LLM-as-judge against a rubric on a sampled slice
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.