How do you detect quality drift in a production LLM system without ground-truth labels?

**Quality drift** is when a deployed LLM feature quietly gets worse over time, even though nobody changed the code. In production you usually cannot grade every answer, because you have no real-time correct answer to compare against. So you watch indirect clues instead. The cheapest clues are user behaviors. A drop in thumbs-up rate, a rise in users asking follow-up questions, or a spike in the model refusing to answer all hint that something changed. These are free and real-time, but they are noisy and can lag the actual problem. A stronger technique is a frozen canary set: a fixed list of test queries you save once and never change. You run them through an LLM judge on a schedule and watch whether the score moves. Because the inputs never change, a score drop points at the model or the system, not at your users. Finally, you watch the incoming queries themselves. If users start asking very different questions than before, the model may be seeing inputs it was never good at. That is input drift, and it is different from the model itself degrading. The key skill is separating these causes, because each one has a different fix.

Without labels, you cannot measure accuracy directly. So you instrument three separate things and avoid conflating them. **Input drift.** Track the distribution of incoming queries. Bucket query embeddings or lexical features, then run a distance test on the histogram. PSI and KL divergence both flag when this week's traffic diverges from a baseline window. This tells you users changed, not that quality dropped. **Model or system drift.** Keep a frozen canary set: a fixed, version-controlled set of representative inputs. Re-run it on a schedule through an LLM judge whose model version and temperature are pinned. Because both inputs and judge are constant, a score move isolates a change in the model, prompt, or retrieval layer. **Judge drift.** The judge is itself a model. Pin its version and snapshot its scores on a tiny gold subset. If the vendor silently updates it, your canary scores shift for the wrong reason. Layered on top are live proxy signals: feedback rate, follow-up rate, and refusal rate. These are real-time and cheap. But they lag real quality, are gameable through UI changes, and conflate root causes. The discipline is triangulation. Proxies are a fast alarm, the canary is the controlled measurement, and distance tests attribute the cause.