LLM-as-judge scores drift up across 50k weekly evals while user complaints hold steady. Diagnose likely causes and design a calibration system preventing silent score inflation.

Short answer·Hard·4.0 · 0·~3 min·Relevant atDatabricks

Attempt it

Free · 2 AI evals / day

When LLM-as-judge scores drift upward but user complaints stay flat, the judge is becoming miscalibrated. Three common causes. First, the judge model was silently updated by the provider. API-served models change without notice, and a new version might score more generously. Second, the judge's sense of what a "4" means shifts as the outputs it evaluates get gradually better or change style. Third, the evaluated model's outputs get longer over time, and LLM judges have a known bias toward scoring verbose responses higher. **The calibration system** has five parts. Pin the judge model to a specific version so provider updates cannot silently shift scores. Maintain a frozen set of 200 pre-scored examples with human labels and run the judge against them weekly; if agreement drops, the judge has drifted. Include sentinel outputs (deliberately terrible and excellent) as canary checks. Run multiple judges from different model families and flag disagreements. Track the full score distribution over time and alert on mean drift, variance compression, or mode collapse. The key insight is that calibrating the judge is its own eval problem. The calibration set is the judge's golden set, and the agreement metric is the judge's quality score.

**Diagnosis.** Three causes, often compounding. (1) Silent judge model update: the provider changes the API-served model, shifting scoring behavior. (2) Rubric anchoring decay: as the evaluated outputs improve or change style, the judge recalibrates its internal anchor upward. (3) Verbosity bias amplification: outputs get longer over time, and the judge scores length as quality. **Calibration system.** Five mechanisms. (a) Pin the judge to a specific model version; never use a "latest" alias. (b) Frozen calibration set: 200 pre-scored examples covering the full score range, with human ground truth. Run weekly; compute Cohen's kappa. Alert if kappa drops below 0.6. (c) Sentinel pairs: 10 to 20 outputs engineered for unambiguous scores of 1 and 5. If sentinels drift, the judge is catastrophically miscalibrated. (d) Multi-judge ensemble: 2 to 3 judges from different model families (Claude Opus 4.7, GPT-5.5). Divergence flags examples for human review. (e) Score distribution monitoring: alert on mean drift, variance compression, mode collapse, floor erosion, ceiling accumulation. **Meta-insight.** Calibrating the judge is itself an eval problem. The calibration set is the judge's golden set, the agreement metric is its quality score, and the weekly run is its regression gate. Every eval principle applies recursively.

Quick reference

TL;DR

Score inflation comes from silent model updates, rubric anchoring decay, and verbosity bias. Fix with pinned versions, a frozen calibration set, sentinel pairs, multi-judge ensemble, and distribution monitoring.

Memory aid

Easy to grasp

Imagine a teacher who grades 500 essays a week. Over time the essays get a little longer and a little more polished, so the teacher starts giving higher grades without realizing it. Meanwhile, when you ask the students questions in person, they are no better than before. The grades drifted because the teacher's sense of what counts as an A shifted as the average essay improved. To catch this, you slip in the same 10 test essays every week. If the teacher starts grading those test essays higher than last month, you know the grading standard has shifted. You also have two other teachers grade a random sample; if they disagree, someone's calibration is off.

Key concepts

LLM-as-Judge Calibration

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Open by framing the problem: scores up, complaints steady, therefore the measurement instrument is drifting. Diagnose three causes in order: silent model update, rubric anchoring decay, verbosity bias amplification. Then design the five-part calibration system: pinned versions, frozen calibration set with weekly kappa, sentinel pairs, multi-judge ensemble, distribution monitoring. Close with the meta-insight that judge calibration is a recursive eval problem.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's research team uses pinned model versions for LLM-as-judge evaluations, with re-calibration runs on a frozen human-labeled set when migrating between Claude model versions.
Braintrust supports multi-judge scoring with automatic flagging of inter-judge disagreements, enabling teams to detect drift across judge model families.
Arize Phoenix provides score distribution monitoring dashboards with configurable alerts on mean shift, variance changes, and mode collapse across evaluation runs.
DeepEval includes built-in support for sentinel-style test cases with fixed expected scores, used as regression canaries in CI pipelines.
Production teams at scale in 2026 commonly run weekly calibration checks against 100 to 300 human-labeled examples, treating kappa below 0.6 as a halt and recalibrate signal.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QThe calibration set itself is 18 months old. How do you know it still represents the current task distribution?

QYour multi-judge ensemble has two judges that agree with each other but both disagree with human labels. What happened?

QHow would you handle the case where the calibration set kappa is fine but users are still complaining?

QAt $5 per million input tokens for Claude Opus 4.7, what is the weekly cost of running 50,000 examples through a 3-judge ensemble, and how do you manage it?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming score inflation means the product is improving. When user complaints are steady but eval scores rise, the judge is drifting, not the product. Treating inflated scores as real leads to false confidence and missed regressions.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.