LLM-as-judge scores drift up across 50k weekly evals while user complaints hold steady. Diagnose likely causes and design a calibration system preventing silent score inflation.

Short answer·Hard·4.0 · 0·~3 min·Asked atOla Redis Sourcegraph·Relevant atAnyscale Databricks Datadog Scale Ai

Attempt it

Free · 2 AI evals / day

When LLM-as-judge scores drift upward but user complaints stay flat, the judge is becoming miscalibrated. Three common causes. First, the judge model was silently updated by the provider. API-served models change without notice, and a new version might score more generously. Second, the judge's sense of what a "4" means shifts as the outputs it evaluates get gradually better or change style. Third, the evaluated model's outputs get longer over time, and LLM judges have a known bias toward scoring verbose responses higher. **The calibration system** has five parts. Pin the judge model to a specific version so provider updates cannot silently shift scores. Maintain a frozen set of 200 pre-scored examples with human labels and run the judge against them weekly; if agreement drops, the judge has drifted. Include sentinel outputs (deliberately terrible and excellent) as canary checks. Run multiple judges from different model families and flag disagreements. Track the full score distribution over time and alert on mean drift, variance compression, or mode collapse. The key insight is that calibrating the judge is its own eval problem. The calibration set is the judge's golden set, and the agreement metric is the judge's quality score.

**Diagnosis.** Three causes, often compounding. (1) Silent judge model update: the provider changes the API-served model, shifting scoring behavior. (2) Rubric anchoring decay: as the evaluated outputs improve or change style, the judge recalibrates its internal anchor upward. (3) Verbosity bias amplification: outputs get longer over time, and the judge scores length as quality. **Calibration system.** Five mechanisms. (a) Pin the judge to a specific model version; never use a "latest" alias. (b) Frozen calibration set: 200 pre-scored examples covering the full score range, with human ground truth. Run weekly; compute Cohen's kappa. Alert if kappa drops below 0.6. (c) Sentinel pairs: 10 to 20 outputs engineered for unambiguous scores of 1 and 5. If sentinels drift, the judge is catastrophically miscalibrated. (d) Multi-judge ensemble: 2 to 3 judges from different model families (Claude Opus 4.7, GPT-5.5). Divergence flags examples for human review. (e) Score distribution monitoring: alert on mean drift, variance compression, mode collapse, floor erosion, ceiling accumulation. **Meta-insight.** Calibrating the judge is itself an eval problem. The calibration set is the judge's golden set, the agreement metric is its quality score, and the weekly run is its regression gate. Every eval principle applies recursively.

TL;DR

Score inflation comes from silent model updates, rubric anchoring decay, and verbosity bias. Fix with pinned versions, a frozen calibration set, sentinel pairs, multi-judge ensemble, and distribution monitoring.

Memory aid

Easy to grasp

Imagine a teacher who grades 500 essays a week. Over time the essays get a little longer and a little more polished, so the teacher starts giving higher grades without realizing it. Meanwhile, when you ask the students questions in person, they are no better than before. The grades drifted because the teacher's sense of what counts as an A shifted as the average essay improved. To catch this, you slip in the same 10 test essays every week. If the teacher starts grading those test essays higher than last month, you know the grading standard has shifted. You also have two other teachers grade a random sample; if they disagree, someone's calibration is off.

Key concepts

LLM-as-Judge Calibration

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

LLM-as-judge is the dominant approach for evaluating open-ended LLM outputs at scale in 2026. It works well when calibrated, but it has a fundamental vulnerability: the judge is itself a model, and models drift. When scores inflate silently while user-perceived quality stays flat, the eval system has become a false reassurance engine.

The scenario in this question is common in production: 50,000 evaluations per week, scores trending upward over months, user complaints holding steady. The divergence between eval signal and user signal is the symptom. The causes are predictable: silent judge model updates, rubric anchoring decay, and bias amplification. The solution is a calibration system that treats the judge as a measurement instrument requiring regular re-calibration, not a fixed oracle.

This walkthrough diagnoses the three drift sources, designs a five-part calibration system, and closes with the recursive insight that calibrating the judge is itself an eval problem.

Diagnosing score inflation: three compounding causes

Silent judge model update is the most common cause of sudden score drift. LLM providers serve models through APIs, and many providers update the model behind an alias (such as a "latest" or default endpoint) without explicit notice. A model version change can shift scoring behavior in any direction: a new version might be more generous, more sensitive to fluency cues, or more reluctant to assign low scores. If the judge endpoint points to a model alias rather than a pinned version, every provider update is a potential score shift that the eval team never authorized.

The diagnostic signature of a silent update is a step change in scores over a short window (days, not weeks), often coinciding with a provider announcement about model improvements. If you track the weekly mean score and see a discontinuity, check the provider's changelog.

Rubric anchoring decay is the slow variant. The judge's interpretation of what constitutes a score of 4 versus 5 drifts as the distribution of outputs it evaluates changes over time. If the model being evaluated gradually improves (through fine-tuning, prompt engineering, or RLHF updates), the judge sees better outputs more often. Its implicit calibration anchor shifts upward. A response that would have scored 3 six months ago now scores 4 because the judge's reference frame has changed. Scores inflate not because the judge model changed, but because the input distribution to the judge shifted.

Verbosity and position bias amplification is the interaction effect. LLM judges have well-documented biases: they tend to score longer responses higher (verbosity bias) and to weight content at certain positions more heavily (position bias, also called primacy or recency bias). As the evaluated model's output style evolves, often toward longer, more detailed responses driven by RLHF optimization for user engagement, the judge's verbosity bias becomes a larger component of the total score. Scores go up because the outputs got longer, not because the underlying quality improved.

Calibration mechanism one: version pinning and the frozen calibration set

Calibration mechanism two: sentinel pairs and multi-judge ensembles

Calibration mechanism three: score distribution monitoring

The recursive insight and operational discipline

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's research team uses pinned model versions for LLM-as-judge evaluations, with re-calibration runs on a frozen human-labeled set when migrating between Claude model versions.
Braintrust supports multi-judge scoring with automatic flagging of inter-judge disagreements, enabling teams to detect drift across judge model families.

What an interviewer would ask next. Try answering before peeking at the approach.

QThe calibration set itself is 18 months old. How do you know it still represents the current task distribution?

The calibration set must be refreshed on the same cadence as the product's golden set. Compare the embedding distribution of the calibration examples to recent production outputs. If they diverge, sample new calibration examples from production, have humans label them, and merge into the set. Old examples provide longitudinal comparability; new examples provide distribution relevance.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming score inflation means the product is improving. When user complaints are steady but eval scores rise, the judge is drifting, not the product. Treating inflated scores as real leads to false confidence and missed regressions.

60 second bullets to scan on the way to the call.

Name silent model update as a cause of score drift and explain why version pinning prevents it
Name rubric anchoring decay and explain how the output distribution shift causes the judge to recalibrate

Primary sources. Browse if you want the original framing.