Your LLM judge gives everything a 4 out of 5. Your colleague says the judge is not calibrated. What does calibration mean here?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Your answer

When your LLM judge gives everything a 4 out of 5, it is producing numbers, but those numbers are not useful. They do not distinguish good outputs from bad ones. Your colleague is saying the judge is not calibrated: its scores do not match what a human expert would assign. **Calibration** in this context means the judge's scores align with human expert scores on the same items. A calibrated judge gives high scores to outputs that humans rate highly and low scores to outputs that humans rate poorly. An uncalibrated judge might cluster all scores in a narrow range, systematically overrate one type of output, or fail to use the full scale. You check calibration by running the judge on a subset of outputs where you already have human expert labels. Then you compute agreement: Cohen's kappa for categorical scores, Pearson correlation for continuous scores. If agreement is low, the judge needs adjustment. Common fixes include refining the rubric (more explicit criteria, better anchor examples), adjusting the eval prompt template (adding 'be strict' or 'use the full range'), or switching to a different judge model. After each change, you re-run the calibration check to see if agreement improved.

**Calibration** means the LLM judge's scores match human expert scores on the same items. When a judge gives everything a 4 out of 5, it is uncalibrated: the scores carry no discriminative signal, even if they are internally consistent. The calibration check is straightforward. Collect human expert labels on 50 to 100 representative outputs. Run the LLM judge on the same outputs. Compute agreement: Cohen's kappa for ordinal scales, Pearson correlation for continuous scores. A kappa above 0.6 is the common threshold for 'acceptable agreement.' When calibration fails, the fix targets one of three root causes. Score clustering (everything is 4) usually means the rubric lacks clear differentiation between score levels; adding anchor examples at each level helps. Systematic bias (the judge overrates verbose responses) means the eval prompt template needs explicit instructions to ignore length. Dimension confusion (the judge scores tone when it should score accuracy) means the rubric dimensions are not separated clearly enough. Calibration is not a one-time check. It degrades when the judge model version changes, when the input distribution shifts, or when the rubric is updated. Periodic re-calibration, running the judge on fresh human-labeled samples, is essential to maintain trust in the evaluation signal.

Quick reference

TL;DR

Calibration means the LLM judge's scores align with human expert scores on the same items. Check it by computing agreement (Cohen's kappa or correlation) on a labeled subset.

Memory aid

Easy to grasp

Imagine a kitchen thermometer that always reads 180 degrees no matter what you put it in. Cold water? 180. Boiling soup? 180. The thermometer works (it gives a number) but it is not calibrated (the number does not reflect reality). To calibrate it, you check it against a known standard: stick it in ice water (should read 0) and boiling water (should read 100), then adjust until it matches. An LLM judge is the same. It gives scores, but the scores might not reflect what a human expert would say. To calibrate, you give the judge outputs that humans have already graded and check if the scores match. If they do not, you adjust the rubric, add anchor examples, or try a different judge model until they do.

Key concepts

LLM-as-judge calibration LLM-as-judge Eval fundamentals

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define calibration, explain why score clustering is a red flag, describe the calibration check with kappa, list common fixes, and close with the continuous re-calibration discipline.

Real-world examples

Real products, models, and research that use this idea.

MT-Bench validated its LLM judge by comparing judge scores to human expert scores on a subset and reporting correlation metrics.
Braintrust AI lets teams define calibration sets and automatically computes judge-human agreement when the eval prompt changes.
AlpacaEval 2 reports the correlation between its LLM judge's win rates and Chatbot Arena human rankings as a calibration benchmark.
Google DeepMind's evaluation teams periodically re-calibrate their LLM judges against fresh human labels when switching model versions.
Promptfoo supports a calibration mode where you provide human labels and the tool reports agreement metrics per evaluation dimension.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow many human-labeled items do you need for a reliable calibration check?

QWhat is the difference between calibration and inter-rater reliability?

QHow do you handle calibration when human experts themselves disagree?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Deploying an LLM judge without checking calibration against human labels. The judge might produce consistent scores that are consistently wrong.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.