LLM-as-judge scores drift up across 50k weekly evals while user complaints hold steady. Diagnose likely causes and design a calibration system preventing silent score inflation.
Short answer·Hard·4.0 · 0·~3 min·Relevant atDatabricks
LLM-as-judge scores drift up across 50k weekly evals while user complaints hold steady. Diagnose likely causes and design a calibration system preventing silent score inflation.
Free · 2 AI evals / day
When LLM-as-judge scores drift upward but user complaints stay flat, the judge is becoming miscalibrated. Three common causes.
First, the judge model was silently updated by the provider. API-served models change without notice, and a new version might score more generously. Second, the judge's sense of what a "4" means shifts as the outputs it evaluates get gradually better or change style. Third, the evaluated model's outputs get longer over time, and LLM judges have a known bias toward scoring verbose responses higher.
**The calibration system** has five parts. Pin the judge model to a specific version so provider updates cannot silently shift scores. Maintain a frozen set of 200 pre-scored examples with human labels and run the judge against them weekly; if agreement drops, the judge has drifted. Include sentinel outputs (deliberately terrible and excellent) as canary checks. Run multiple judges from different model families and flag disagreements. Track the full score distribution over time and alert on mean drift, variance compression, or mode collapse.
The key insight is that calibrating the judge is its own eval problem. The calibration set is the judge's golden set, and the agreement metric is the judge's quality score.
**Diagnosis.** Three causes, often compounding. (1) Silent judge model update: the provider changes the API-served model, shifting scoring behavior. (2) Rubric anchoring decay: as the evaluated outputs improve or change style, the judge recalibrates its internal anchor upward. (3) Verbosity bias amplification: outputs get longer over time, and the judge scores length as quality.
**Calibration system.** Five mechanisms. (a) Pin the judge to a specific model version; never use a "latest" alias. (b) Frozen calibration set: 200 pre-scored examples covering the full score range, with human ground truth. Run weekly; compute Cohen's kappa. Alert if kappa drops below 0.6. (c) Sentinel pairs: 10 to 20 outputs engineered for unambiguous scores of 1 and 5. If sentinels drift, the judge is catastrophically miscalibrated. (d) Multi-judge ensemble: 2 to 3 judges from different model families (Claude Opus 4.7, GPT-5.5). Divergence flags examples for human review. (e) Score distribution monitoring: alert on mean drift, variance compression, mode collapse, floor erosion, ceiling accumulation.
**Meta-insight.** Calibrating the judge is itself an eval problem. The calibration set is the judge's golden set, the agreement metric is its quality score, and the weekly run is its regression gate. Every eval principle applies recursively.
Quick reference
TL;DR
Score inflation comes from silent model updates, rubric anchoring decay, and verbosity bias. Fix with pinned versions, a frozen calibration set, sentinel pairs, multi-judge ensemble, and distribution monitoring.
Imagine a teacher who grades 500 essays a week. Over time the essays get a little longer and a little more polished, so the teacher starts giving higher grades without realizing it. Meanwhile, when you ask the students questions in person, they are no better than before. The grades drifted because the teacher's sense of what counts as an A shifted as the average essay improved. To catch this, you slip in the same 10 test essays every week. If the teacher starts grading those test essays higher than last month, you know the grading standard has shifted. You also have two other teachers grade a random sample; if they disagree, someone's calibration is off.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A small eval set run manually once a month with human scoring. At low volume with human scorers, the drift dynamics described here do not apply. Human inter-annotator agreement has its own issues but does not suffer from silent model updates or verbosity bias in the same way.
Deterministic metric evaluation (BLEU, ROUGE, exact match) with no LLM judge involved. Deterministic metrics do not drift. The calibration system addresses model-based scoring specifically. If the eval uses only automated metrics, the investment in judge calibration is unnecessary.
A prototype where the eval is run once to pick a model and then discarded. Calibration systems address drift over time. A one-shot eval has no time dimension, so the calibration overhead is not justified.
Quick scan · 2–4 min
Everything important, quickly.
Open by framing the problem: scores up, complaints steady, therefore the measurement instrument is drifting. Diagnose three causes in order: silent model update, rubric anchoring decay, verbosity bias amplification. Then design the five-part calibration system: pinned versions, frozen calibration set with weekly kappa, sentinel pairs, multi-judge ensemble, distribution monitoring. Close with the meta-insight that judge calibration is a recursive eval problem.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's research team uses pinned model versions for LLM-as-judge evaluations, with re-calibration runs on a frozen human-labeled set when migrating between Claude model versions.
2Braintrust supports multi-judge scoring with automatic flagging of inter-judge disagreements, enabling teams to detect drift across judge model families.
3Arize Phoenix provides score distribution monitoring dashboards with configurable alerts on mean shift, variance changes, and mode collapse across evaluation runs.
4DeepEval includes built-in support for sentinel-style test cases with fixed expected scores, used as regression canaries in CI pipelines.
5Production teams at scale in 2026 commonly run weekly calibration checks against 100 to 300 human-labeled examples, treating kappa below 0.6 as a halt and recalibrate signal.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QThe calibration set itself is 18 months old. How do you know it still represents the current task distribution?
Sign in or go Pro to see the approach for this follow-up.
QYour multi-judge ensemble has two judges that agree with each other but both disagree with human labels. What happened?
Sign in or go Pro to see the approach for this follow-up.
QHow would you handle the case where the calibration set kappa is fine but users are still complaining?
Sign in or go Pro to see the approach for this follow-up.
QAt $5 per million input tokens for Claude Opus 4.7, what is the weekly cost of running 50,000 examples through a 3-judge ensemble, and how do you manage it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming score inflation means the product is improving. When user complaints are steady but eval scores rise, the judge is drifting, not the product. Treating inflated scores as real leads to false confidence and missed regressions.