A production model has no ground-truth labels on live traffic. Design a monitoring plan that catches degradation before users complain.

Short answer·Medium·4.0 · 0·~3 min·Relevant atDatabricks Microsoft

Attempt it

Free · 2 AI evals / day

When your model is in production without ground-truth labels, you need proxy signals that detect quality drops before users complain in support tickets. The first layer is user behavioral signals. Track thumbs-down rate, follow-up question rate (a proxy for "the answer was not helpful enough"), and session abandonment. These are lagging and sparse, but they catch problems severe enough to change how users interact. The second layer is LLM-as-judge. Run a powerful judge model on 1 to 5 percent of production conversations daily, scoring each against a rubric. This catches subtle quality drops (increased hedging, factual drift) that users tolerate but that erode trust over time. The third layer is a canary eval. Keep a fixed golden set of 50 to 100 examples and run it through the production endpoint on every deploy. Because the input is fixed, any score change points to the model or infrastructure, not to traffic shifts. The fourth layer is input distribution monitoring. Embed incoming queries and track drift from the golden-set distribution. When traffic shifts, your eval coverage becomes stale and you need to update. Each layer covers a gap the others miss. Together they form an early-warning system.

**Production monitoring without labels** requires layering four proxy signals, because no single signal catches every failure mode. First, user behavioral signals: thumbs-down rate, follow-up question rate, session abandonment. Alert when any moves more than 2 standard deviations from the trailing 7-day baseline. These are lagging but catch severe problems. Second, LLM-as-judge on a rolling sample. Run Claude Opus 4.7 or GPT-5.5 on 1 to 5 percent of daily traffic, scoring against a product-aligned rubric. This catches subtle quality drops (hedging, factual drift, loss of specificity) that users tolerate but that erode trust. Third, canary eval. A fixed golden set (50 to 100 examples) runs through the production endpoint daily or on every deploy. Fixed input means any score change is attributable to the model or infrastructure, isolating model-side regressions from traffic-side shifts. Fourth, input distribution monitoring. Embed incoming queries and track cosine similarity to the golden-set centroid. When production traffic drifts, the eval scores lose representativeness. Alert when drift exceeds a threshold so the team knows to refresh the golden set. Layering matters because each signal has blind spots. User signals miss subtle drops. The judge misses distribution shift. The canary misses novel query types. Distribution monitoring catches when the eval itself goes stale.

Quick reference

TL;DR

Layer four signals: user behavioral proxies, LLM-as-judge on a rolling sample, canary eval on a fixed golden set, and input distribution drift monitoring. Each catches what the others miss.

Memory aid

Easy to grasp

Imagine running a restaurant where you cannot taste every dish. You set up four ways to catch bad food before customers leave angry. First, watch body language: if more people push their plates away, something is wrong. Second, have a food critic taste a few random dishes each day. Third, order the same five dishes every morning and compare them to yesterday's. Fourth, check if your ingredients changed, because new suppliers might mean new problems even if the food critic has not tasted their dishes yet. No single check catches everything, but together they cover the gaps.

Key concepts

Production Monitoring

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Walk the four layers in order: user behavioral signals (lagging, sparse), LLM-as-judge on rolling sample (workhorse), canary eval on fixed golden set (isolation layer), input distribution monitoring (representativeness check). For each, name what it catches and its blind spot. Explain why layering is necessary. Close with alert thresholds and the separation between detection (automated) and diagnosis (human).

Real-world examples

Real products, models, and research that use this idea.

Anthropic monitors Claude production quality using layered signals including automated judge scoring on sampled traffic and canary evaluations on fixed test sets with alerting on score regressions.
Google Gemini's production monitoring pipeline tracks embedding drift on incoming queries to detect when the traffic distribution diverges from the evaluation coverage.
LangSmith and Weights and Biases provide production monitoring dashboards that combine user feedback signals, LLM-as-judge scores, and latency metrics for deployed LLM applications.
Datadog and New Relic have added LLM-specific monitoring features that track quality proxies (hallucination rate, refusal rate) alongside standard operational metrics for production LLM services.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour LLM-as-judge scores have been stable for a month, but user thumbs-down rate just spiked. What could explain the divergence?

QRunning the judge on 5 percent of traffic costs $500 per day. How do you reduce cost without losing signal?

QYour canary eval shows a 5-point drop on the morning deploy but recovers by evening. What happened?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Relying on a single proxy signal (e.g. only thumbs-down rate) which is too sparse and lagging to catch subtle quality degradation before it affects user retention.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.