Complete: Cohen's kappa >___ is 'substantial' agreement; >___ is 'almost perfect' agreement

Attempt it

When calibrating an LLM judge against human raters, Cohen's kappa > is considered 'substantial' agreement, and kappa > is considered 'almost perfect' agreement. Below , the judge is likely too unreliable for production use.

Quick reference

TL;DR

Cohen's kappa above 0.6 signals substantial judge-human agreement; above 0.8 is almost perfect. Below 0.6 the judge is usually too unreliable to ship.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine two teachers grading the same stack of essays pass or fail. If they agree on 90 percent of essays, that sounds great. But what if both just guess and most essays happen to pass? They would still agree most of the time by pure luck. Cohen's kappa is a fairness-adjusted agreement score. It asks how much two graders agree BEYOND what random guessing would give. A kappa of 0 means they agree no better than coin flips. A kappa of 1 means they agree perfectly. When you swap one teacher for an LLM judge, you want kappa above 0.6 before you trust it, and above 0.8 before you call it as good as a human.

Key concepts

LLM-as-judge calibration Cohen's kappa Inter-rater agreement

Quick scan · 2–4 min

Everything important, quickly.

5 min: the kappa formula and why chance correction matters, the Landis-Koch bands, the kappa paradox on skewed sets, weighted vs Fleiss variants, and the production calibration loop with human holdouts.

Side-by-side

Kappa range	Landis-Koch label	Production read
< 0.0	Poor (worse than chance)	Broken judge or rubric
0.0 to 0.20	Slight	Unusable
0.21 to 0.40	Fair	Weak, diagnose by class
0.41 to 0.60	Moderate	Coarse pre-filter only
0.61 to 0.80	Substantial	Minimum production bar
0.81 to 1.00	Almost perfect	Gold standard, human-equivalent

Real-world examples

Real products, models, and research that use this idea.

RAGAS calibration guides recommend measuring judge-human kappa on a labelled holdout before trusting faithfulness scores in production.
Anthropic and OpenAI eval cookbooks report inter-rater agreement when validating LLM-judge prompts against human annotators.
LangSmith and Braintrust let teams attach human labels to traces, then surface judge versus human agreement as a calibration signal.
Scale AI and Surge AI annotation pipelines report Cohen's and Fleiss's kappa across human raters as a data-quality gate.
Patronus and Galileo position recurring judge recalibration against human labels as a 2026 production best practice.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can a judge post 95 percent accuracy yet score a kappa near zero?

Sign in or go Pro to see the approach for this follow-up.

QHow would you adapt the metric for a 1 to 5 quality scale instead of pass or fail?

Sign in or go Pro to see the approach for this follow-up.

QA judge has acceptable kappa but ranks two candidate systems opposite to humans. What happened?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.