Calibration means the LLM judge's scores align with human expert scores on the same items. Check it by computing agreement (Cohen's kappa or correlation) on a labeled subset.
Imagine a kitchen thermometer that always reads 180 degrees no matter what you put it in. Cold water? 180. Boiling soup? 180. The thermometer works (it gives a number) but it is not calibrated (the number does not reflect reality). To calibrate it, you check it against a known standard: stick it in ice water (should read 0) and boiling water (should read 100), then adjust until it matches. An LLM judge is the same. It gives scores, but the scores might not reflect what a human expert would say. To calibrate, you give the judge outputs that humans have already graded and check if the scores match. If they do not, you adjust the rubric, add anchor examples, or try a different judge model until they do.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An LLM judge that gives everything a 4 out of 5 is not useful, even if it is fast, consistent, and cheap. The scores carry no signal. Calibration is the property that makes judge scores meaningful: the scores reflect the same quality distinctions that human experts would make.
This deep dive covers what calibration means, how to check it, common failure patterns, and the ongoing discipline of keeping a judge calibrated as conditions change.
What calibration means in practice
A calibrated judge produces scores that match human expert scores on the same items. When a human gives an output a 2, a calibrated judge also gives it a 2 (or close to it). When a human gives a 5, the judge gives a 5. The scores track the same quality distinctions.
An uncalibrated judge breaks this correspondence. The most common failure pattern is score clustering: the judge assigns most outputs the same score (typically in the 3 to 4 range), failing to distinguish good from great or mediocre from bad. Other patterns include systematic overrating (everything gets 4 or 5), systematic underrating (everything gets 1 or 2), and dimension confusion (the judge scores the wrong quality dimension, like rating tone when asked about accuracy).
Calibration is checked empirically. You need a calibration set: 50 to 100 outputs where human experts have assigned scores using the same rubric the judge uses. You run the judge on this set and compute agreement.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- MT-Bench validated its LLM judge by comparing judge scores to human expert scores on a subset and reporting correlation metrics.
- Braintrust AI lets teams define calibration sets and automatically computes judge-human agreement when the eval prompt changes.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow many human-labeled items do you need for a reliable calibration check?
50 to 100 items is the practical minimum. Below 30, agreement metrics are unstable. Above 200, you get diminishing returns. The items should be representative of the distribution the judge will evaluate in production.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Deploying an LLM judge without checking calibration against human labels. The judge might produce consistent scores that are consistently wrong.
60 second bullets to scan on the way to the call.
Define calibration as the judge's scores matching human expert scores
Explain why score clustering (everything is 4) signals miscalibration
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.