How do you measure whether an LLM judge is well calibrated against human raters?
Describe how you would measure the calibration of an LLM judge against human raters. What metric(s) would you use, and what threshold would indicate the judge is suitable for production use?
Validate the judge against human labels with Cohen's kappa, not raw percent agreement, because kappa corrects for chance. Aim for kappa above 0.6 (substantial) on 100-300 paired examples, checked per slice.
Imagine two people guessing whether photos show a cat or a dog, where 90 percent of the photos happen to be cats. If both just shout 'cat' every time, they agree 90 percent of the time, which sounds amazing but proves nothing, because they would agree that often even guessing blindly. Cohen's kappa fixes this. It asks how much they agree beyond what dumb luck alone would produce. Validating an LLM judge against humans is the same idea. The judge and a human both score the same answers, and you measure agreement after subtracting the lucky-guess baseline. A kappa near 1 means the judge truly tracks human judgment. A kappa near 0 means it is no better than a coin flip dressed up as a big agreement number.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An LLM judge is only useful if it agrees with the humans whose judgment it is standing in for. Calibration is the act of proving that agreement with a number you can defend in a design review. The instinct is to grade a batch of outputs with both the judge and a human, count how often they match, and report that as a percentage. That instinct is wrong, and understanding why is the whole point of this question.
The stakes are concrete. Once a judge gates a release, scores an A/B test, or fires a regression alert, every downstream decision inherits the judge's error. A miscalibrated judge does not just produce slightly noisy numbers; it can systematically prefer the wrong model, mask a real quality regression, or greenlight an output a human would have rejected. Calibration is the cheap insurance that keeps those errors from compounding silently across a quarter of dashboards.
The trap is chance. On any real eval set the labels are imbalanced, and high raw agreement can come almost entirely from both raters voting the common label. The fix is to measure agreement above what chance alone would produce, which is exactly what Cohen's kappa computes. This deep dive walks the failure of percent agreement, the kappa correction, the interpretation bands, the data you need to collect, and the slicing discipline that separates a real validation from a vanity metric.
Why raw percent agreement misleads
Suppose your eval task is pass or fail, and 90 percent of candidate answers actually pass. You grade 200 examples with both an LLM judge and a human. The judge agrees with the human on 180 of them, a headline 90 percent agreement that looks like a strong result.
Now imagine a lazy second rater who ignores the content entirely and stamps 'pass' on everything. Against the same 90 percent base rate, that rater also agrees with the human about 90 percent of the time. The number that looked impressive is the number a coin weighted to the majority class would produce.
This is the base-rate problem, and it is not an edge case. Most production eval sets are skewed: most answers are fine, most retrievals are relevant, most responses are safe. The more skewed the labels, the more raw agreement overstates the judge. Any agreement metric that ignores the label distribution is reporting luck as skill.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Metric | When to use | Production threshold |
|---|---|---|
| Raw percent agreement | Never as sole evidence; inflated by chance | Not interpretable alone |
| Cohen's kappa | Categorical or ordinal labels, 2 raters | Above 0.6 substantial |
| Weighted kappa | Ordinal Likert where distance matters | Above 0.6 substantial |
| Fleiss's kappa | 3 or more raters on the same items | Above 0.6 substantial |
| Spearman correlation | Scalar 1 to 5 scores, rank agreement | Rho above 0.7 |
Real products, models, and research that use this idea.
- RAGAS and DeepEval document validating their LLM judges against human-labelled sets with kappa or correlation before teams trust the scores.
- Scale AI and Surge AI report inter-annotator kappa on labelling jobs as a quality gate before labels become ground truth.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can Cohen's kappa drop near 0 even when the judge agrees with humans almost every time?
Walk through the kappa paradox: when one label dominates, expected chance agreement is already very high, so the numerator shrinks. Show the confusion matrix and consider prevalence-adjusted kappa or reporting sensitivity and specificity alongside.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting raw percent agreement as proof the judge works. On an imbalanced label set, two random raters can hit 80 percent agreement, so the number means almost nothing without chance correction.
60 second bullets to scan on the way to the call.
Why raw percent agreement misleads on imbalanced label sets
The kappa formula and what observed versus expected agreement mean
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.