How do you measure whether an LLM judge is well-calibrated against human raters?

Attempt it

Describe how you would measure the calibration of an LLM judge against human raters. What metric(s) would you use, and what threshold would indicate the judge is suitable for production use?

Free · 2 AI evals / day

Quick reference

TL;DR

Validate the judge against human labels with Cohen's kappa, not raw percent agreement, because kappa corrects for chance. Aim for kappa above 0.6 (substantial) on 100-300 paired examples, checked per slice.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine two people guessing whether photos show a cat or a dog, where 90 percent of the photos happen to be cats. If both just shout 'cat' every time, they agree 90 percent of the time, which sounds amazing but proves nothing, because they would agree that often even guessing blindly. Cohen's kappa fixes this. It asks how much they agree beyond what dumb luck alone would produce. Validating an LLM judge against humans is the same idea. The judge and a human both score the same answers, and you measure agreement after subtracting the lucky-guess baseline. A kappa near 1 means the judge truly tracks human judgment. A kappa near 0 means it is no better than a coin flip dressed up as a big agreement number.

Key concepts

LLM-as-judge calibration Cohen's kappa Human evaluation

Quick scan · 2–4 min

Everything important, quickly.

5 min: why percent agreement misleads, the chance-corrected kappa formula, the interpretation bands and production bar, how many labels and raters to collect, per slice checks, and when to switch to weighted kappa or Spearman.

Side-by-side

Metric	When to use	Production threshold
Raw percent agreement	Never as sole evidence; inflated by chance	Not interpretable alone
Cohen's kappa	Categorical or ordinal labels, 2 raters	Above 0.6 substantial
Weighted kappa	Ordinal Likert where distance matters	Above 0.6 substantial
Fleiss's kappa	3 or more raters on the same items	Above 0.6 substantial
Spearman correlation	Scalar 1 to 5 scores, rank agreement	Rho above 0.7

Real-world examples

Real products, models, and research that use this idea.

RAGAS and DeepEval document validating their LLM judges against human-labelled sets with kappa or correlation before teams trust the scores.
Scale AI and Surge AI report inter-annotator kappa on labelling jobs as a quality gate before labels become ground truth.
Chatbot Arena cross-checks its LLM-judge rankings against crowd-sourced human preference votes to confirm the judge tracks humans.
Patronus and Galileo expose judge versus human agreement metrics so customers can see whether an eval judge is calibrated for their domain.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can Cohen's kappa drop near 0 even when the judge agrees with humans almost every time?

Sign in or go Pro to see the approach for this follow-up.

QHow many human labels and raters do you actually need for a trustworthy calibration estimate?

Sign in or go Pro to see the approach for this follow-up.

QWhen would you prefer weighted kappa or Spearman over plain Cohen's kappa?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.