In a deployed LLM system, ground-truth labels are not available in real time. What proxy signals and techniques can you use to detect quality drift, and what are the limitations of each?
Free · 2 AI evals / day
**Quality drift** is when a deployed LLM feature quietly gets worse over time, even though nobody changed the code. In production you usually cannot grade every answer, because you have no real-time correct answer to compare against. So you watch indirect clues instead.
The cheapest clues are user behaviors. A drop in thumbs-up rate, a rise in users asking follow-up questions, or a spike in the model refusing to answer all hint that something changed. These are free and real-time, but they are noisy and can lag the actual problem.
A stronger technique is a frozen canary set: a fixed list of test queries you save once and never change. You run them through an LLM judge on a schedule and watch whether the score moves. Because the inputs never change, a score drop points at the model or the system, not at your users.
Finally, you watch the incoming queries themselves. If users start asking very different questions than before, the model may be seeing inputs it was never good at. That is input drift, and it is different from the model itself degrading.
The key skill is separating these causes, because each one has a different fix.
Without labels, you cannot measure accuracy directly. So you instrument three separate things and avoid conflating them.
**Input drift.** Track the distribution of incoming queries. Bucket query embeddings or lexical features, then run a distance test on the histogram. PSI and KL divergence both flag when this week's traffic diverges from a baseline window. This tells you users changed, not that quality dropped.
**Model or system drift.** Keep a frozen canary set: a fixed, version-controlled set of representative inputs. Re-run it on a schedule through an LLM judge whose model version and temperature are pinned. Because both inputs and judge are constant, a score move isolates a change in the model, prompt, or retrieval layer.
**Judge drift.** The judge is itself a model. Pin its version and snapshot its scores on a tiny gold subset. If the vendor silently updates it, your canary scores shift for the wrong reason.
Layered on top are live proxy signals: feedback rate, follow-up rate, and refusal rate. These are real-time and cheap. But they lag real quality, are gameable through UI changes, and conflate root causes. The discipline is triangulation. Proxies are a fast alarm, the canary is the controlled measurement, and distance tests attribute the cause.
Quick reference
TL;DR
Split drift into three sources: input drift (distance tests on the query stream), model drift (a frozen canary re-judged on a pinned judge), and judge drift. Live proxies catch the rest, but they lag and conflate causes.
Imagine running a kitchen where you never get to taste the food after it leaves. You cannot grade every plate, so you watch for clues. Are customers sending plates back more often (feedback)? Are they asking the waiter to fix things (follow-ups)? Then once a week you cook ten fixed recipes yourself and grade them against your own notes, so you can compare this week to last week fairly (the canary set). You also watch who walks in: if a tour bus of new customers arrives ordering dishes you rarely make, that is a different kind of change than your cooking getting worse. Separating who changed from what changed from who is grading is the whole game.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks with cheap available ground truth, such as code that runs tests or structured extraction Execution-based or exact-match checks give a real label, so proxy drift signals add noise rather than information.
Very low-traffic features where the query stream is too sparse for distance tests PSI and KL need enough samples per bucket to be stable. Sparse traffic produces false drift alarms from noise.
Safety-critical decisions where a drift alarm alone would trigger an irreversible action Proxy and canary signals are noisy and lagging. A human must confirm before any high-stakes rollback or block.
Brand-new features with no baseline window yet collected Drift is defined relative to a baseline. Without a stable reference period there is nothing to compute distance against.
Quick scan · 2–4 min
Everything important, quickly.
6 min: why labels are missing, the three drift sources (input via PSI/KL, model via frozen canary, judge via version pinning), live proxy signals and their limits, and the triangulation pattern that ties them together.
Real-world examples
Real products, models, and research that use this idea.
1Arize and Fiddler ship embedding-drift monitors that compute PSI and KL on production query distributions against a baseline window.
2LangSmith and Langfuse run scheduled canary eval sets through a pinned LLM judge and chart score over time for regression alerts.
3RAGAS faithfulness on a frozen canary set is a common rolling check to catch retrieval or model drift in deployed RAG features.
4Evidently AI provides out of the box PSI and population-distance reports for monitoring feature and embedding drift in production.
5Teams pin judge versions to a specific Claude Opus 4.7 or GPT-5.5 snapshot so a silent vendor model rotation does not corrupt the canary baseline.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy prefer PSI over KL divergence for input drift on a deployed query stream?
Sign in or go Pro to see the approach for this follow-up.
QHow do you separate a real model regression on the canary from judge variance?
Sign in or go Pro to see the approach for this follow-up.
QInput drift fired but canary scores are flat. What does that tell you and what do you do?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Conflating input drift, model drift, and judge drift into one dashboard line. A score drop can mean users changed, the model changed, or the judge version changed, and the fix differs for each.