Complete: Cohen's kappa >___ is 'substantial' agreement; >___ is 'almost perfect' agreement
Cohen's kappa above 0.6 signals substantial judge-human agreement; above 0.8 is almost perfect. Below 0.6 the judge is usually too unreliable to ship.
Imagine two teachers grading the same stack of essays pass or fail. If they agree on 90 percent of essays, that sounds great. But what if both just guess and most essays happen to pass? They would still agree most of the time by pure luck. Cohen's kappa is a fairness-adjusted agreement score. It asks how much two graders agree BEYOND what random guessing would give. A kappa of 0 means they agree no better than coin flips. A kappa of 1 means they agree perfectly. When you swap one teacher for an LLM judge, you want kappa above 0.6 before you trust it, and above 0.8 before you call it as good as a human.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Cohen's kappa is the workhorse metric for deciding whether an LLM judge can stand in for a human grader. The setup is simple: take a held-out set of outputs, have both a human and the judge label each one, and ask how much they agree. The trap is that the obvious metric, raw percent agreement, is badly inflated whenever one label dominates the set.
This question fixes two named thresholds from the 1977 Landis and Koch paper: kappa above 0.6 is substantial agreement, the usual minimum for trusting a judge in production, and kappa above 0.8 is almost perfect, the gold standard where the judge is effectively a second human. The deep dive derives the formula, explains observed and chance agreement, walks the interpretation bands, and covers the failure modes that bite at calibration time.
Why raw percent agreement fails
Suppose a judge and a human both label 200 model outputs as acceptable or not acceptable, and 190 of the outputs really are acceptable. If the judge just labels everything acceptable, it agrees with the human on 190 out of 200, a tidy-looking 95 percent. But the judge has learned nothing; it is exploiting the skew.
This is why raw percent agreement is the wrong metric. It rewards a constant guesser on any imbalanced set, and most real eval sets are imbalanced because good models are usually right. You need a metric that asks a sharper question: how much do the two raters agree beyond what they would hit by chance given how often each one uses each label?
That is exactly what Cohen's kappa was designed to answer. It strips out the chance-level agreement and reports only the surplus, which is why a 95 percent raw agreement can correspond to a kappa near zero.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Kappa range | Landis-Koch label | Production read |
|---|---|---|
| < 0.0 | Poor (worse than chance) | Broken judge or rubric |
| 0.0 to 0.20 | Slight | Unusable |
| 0.21 to 0.40 | Fair | Weak, diagnose by class |
| 0.41 to 0.60 | Moderate | Coarse pre-filter only |
| 0.61 to 0.80 | Substantial | Minimum production bar |
| 0.81 to 1.00 | Almost perfect | Gold standard, human-equivalent |
Real products, models, and research that use this idea.
- RAGAS calibration guides recommend measuring judge-human kappa on a labelled holdout before trusting faithfulness scores in production.
- Anthropic and OpenAI eval cookbooks report inter-rater agreement when validating LLM-judge prompts against human annotators.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can a judge post 95 percent accuracy yet score a kappa near zero?
Walk through the kappa paradox: when one class dominates, the chance term p_e approaches the observed agreement, so the numerator p_o minus p_e collapses toward zero even though raw accuracy looks excellent.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting raw percent agreement instead of kappa. On a skewed label set two raters can agree 90 percent of the time while their chance-adjusted agreement is near zero.
60 second bullets to scan on the way to the call.
The kappa formula and what observed versus chance agreement mean
Why raw percent agreement misleads on skewed label sets
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.