A production model has no ground-truth labels on live traffic. Design a monitoring plan that catches degradation before users complain.
Free · 2 AI evals / day
When your model is in production without ground-truth labels, you need proxy signals that detect quality drops before users complain in support tickets.
The first layer is user behavioral signals. Track thumbs-down rate, follow-up question rate (a proxy for "the answer was not helpful enough"), and session abandonment. These are lagging and sparse, but they catch problems severe enough to change how users interact.
The second layer is LLM-as-judge. Run a powerful judge model on 1 to 5 percent of production conversations daily, scoring each against a rubric. This catches subtle quality drops (increased hedging, factual drift) that users tolerate but that erode trust over time.
The third layer is a canary eval. Keep a fixed golden set of 50 to 100 examples and run it through the production endpoint on every deploy. Because the input is fixed, any score change points to the model or infrastructure, not to traffic shifts.
The fourth layer is input distribution monitoring. Embed incoming queries and track drift from the golden-set distribution. When traffic shifts, your eval coverage becomes stale and you need to update.
Each layer covers a gap the others miss. Together they form an early-warning system.
**Production monitoring without labels** requires layering four proxy signals, because no single signal catches every failure mode.
First, user behavioral signals: thumbs-down rate, follow-up question rate, session abandonment. Alert when any moves more than 2 standard deviations from the trailing 7-day baseline. These are lagging but catch severe problems.
Second, LLM-as-judge on a rolling sample. Run Claude Opus 4.7 or GPT-5.5 on 1 to 5 percent of daily traffic, scoring against a product-aligned rubric. This catches subtle quality drops (hedging, factual drift, loss of specificity) that users tolerate but that erode trust.
Third, canary eval. A fixed golden set (50 to 100 examples) runs through the production endpoint daily or on every deploy. Fixed input means any score change is attributable to the model or infrastructure, isolating model-side regressions from traffic-side shifts.
Fourth, input distribution monitoring. Embed incoming queries and track cosine similarity to the golden-set centroid. When production traffic drifts, the eval scores lose representativeness. Alert when drift exceeds a threshold so the team knows to refresh the golden set.
Layering matters because each signal has blind spots. User signals miss subtle drops. The judge misses distribution shift. The canary misses novel query types. Distribution monitoring catches when the eval itself goes stale.
Quick reference
TL;DR
Layer four signals: user behavioral proxies, LLM-as-judge on a rolling sample, canary eval on a fixed golden set, and input distribution drift monitoring. Each catches what the others miss.
Imagine running a restaurant where you cannot taste every dish. You set up four ways to catch bad food before customers leave angry. First, watch body language: if more people push their plates away, something is wrong. Second, have a food critic taste a few random dishes each day. Third, order the same five dishes every morning and compare them to yesterday's. Fourth, check if your ingredients changed, because new suppliers might mean new problems even if the food critic has not tasted their dishes yet. No single check catches everything, but together they cover the gaps.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A batch processing pipeline where every output is reviewed by a human before delivery. If human review covers 100 percent of outputs, the monitoring plan adds cost without catching anything the human review misses. Invest in reviewer quality instead.
An internal tool with fewer than 50 daily queries where manual spot-checking is feasible. At very low volume, automated monitoring generates more noise than signal. A daily manual review of all queries is cheaper and more effective.
A prototype or demo product with no SLA on quality and no real users depending on it. Monitoring infrastructure has a setup and maintenance cost. For prototypes, invest that effort in offline evaluation and iterate faster.
Quick scan · 2–4 min
Everything important, quickly.
Walk the four layers in order: user behavioral signals (lagging, sparse), LLM-as-judge on rolling sample (workhorse), canary eval on fixed golden set (isolation layer), input distribution monitoring (representativeness check). For each, name what it catches and its blind spot. Explain why layering is necessary. Close with alert thresholds and the separation between detection (automated) and diagnosis (human).
Real-world examples
Real products, models, and research that use this idea.
1Anthropic monitors Claude production quality using layered signals including automated judge scoring on sampled traffic and canary evaluations on fixed test sets with alerting on score regressions.
2Google Gemini's production monitoring pipeline tracks embedding drift on incoming queries to detect when the traffic distribution diverges from the evaluation coverage.
3LangSmith and Weights and Biases provide production monitoring dashboards that combine user feedback signals, LLM-as-judge scores, and latency metrics for deployed LLM applications.
4Datadog and New Relic have added LLM-specific monitoring features that track quality proxies (hallucination rate, refusal rate) alongside standard operational metrics for production LLM services.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYour LLM-as-judge scores have been stable for a month, but user thumbs-down rate just spiked. What could explain the divergence?
Sign in or go Pro to see the approach for this follow-up.
QRunning the judge on 5 percent of traffic costs $500 per day. How do you reduce cost without losing signal?
Sign in or go Pro to see the approach for this follow-up.
QYour canary eval shows a 5-point drop on the morning deploy but recovers by evening. What happened?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Relying on a single proxy signal (e.g. only thumbs-down rate) which is too sparse and lagging to catch subtle quality degradation before it affects user retention.