Cohen's kappa measures how much two human raters agree beyond chance. Above 0.6 is substantial, above 0.8 almost perfect; it sets the reliability ceiling any LLM judge can hope to match.
Imagine two friends grading the same stack of essays as pass or fail. If they agree on 80 out of 100, that sounds great. But suppose almost every essay is a clear pass, so even two people guessing 'pass' every time would agree most of the time. Raw agreement is fooled by easy cases. Kappa is a smarter score: it asks how much the two friends agree above what pure luck would give. A kappa near zero means they were basically guessing in lockstep. A kappa near one means they truly see eye to eye on the hard calls too. In LLM work, you first check that humans agree with each other before you trust any automated grader to match them.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Cohen's kappa is the standard way to measure whether two human raters agree with each other beyond what chance alone would produce. In LLM evaluation it shows up everywhere humans label outputs: rating helpfulness, judging factual correctness, scoring safety, or building the gold sets that automated judges are later validated against.
The reason it matters is subtle but important. Raw percent agreement is easy to compute and easy to misread, because on skewed tasks two raters can agree most of the time just by both favouring the common answer. Kappa strips out that lucky overlap and reports the agreement that reflects real, shared judgment. It is also the quantity that sets the reliability ceiling for any LLM-as-judge you might later deploy.
Think of human annotation as the foundation the whole eval stack rests on. You write a rubric, you have people apply it, and you turn their labels into a gold set. Every downstream number, including the score your automated judge reports, inherits whatever consistency that human labelling had. Kappa is how you put a number on that consistency before you build anything on top of it.
What kappa actually measures
Cohen's kappa compares two quantities: how often the raters actually agreed, and how often they would be expected to agree by chance given how often each rater uses each label. The chance term is computed from the marginal rates, not assumed to be fifty-fifty. That detail is the whole point. If a rater labels 'good' nine times out of ten, then 'both said good' is cheap and should not count as strong agreement.
The formula is compact:
The numerator is observed agreement minus chance agreement. The denominator is the maximum possible improvement over chance, namely one minus the chance term. So kappa answers a clean question: of all the agreement that was not just luck, how much did the raters capture? It rescales the leftover agreement onto a 0-to-1 line.
A kappa of 1 means they agreed on everything. A kappa of 0 means they did no better than guessing from their own label habits. Negative values mean they agreed less than chance, which usually signals a broken rubric, swapped labels, or two raters who interpreted the task in opposite ways. In practice you rarely see negatives outside data-entry bugs, but when you do, stop and inspect the raw confusion table rather than trusting the single summary number.
\kappa = \frac{p_o - p_e}{1 - p_e}Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic and OpenAI both double-label safety eval sets and report inter-annotator kappa before treating labels as ground truth.
- RAGAS and LangSmith documentation recommend validating an LLM judge against a human-labelled holdout using kappa or correlation.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can two raters show 90 percent agreement yet a kappa near zero?
Work through a skewed binary task where one class dominates. Compute expected chance agreement from the marginal rates, then plug into the kappa formula and watch the numerator collapse.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting raw percent agreement instead of kappa. High agreement on a skewed label set can look strong while raters are effectively guessing in lockstep with the majority class.
60 second bullets to scan on the way to the call.
What kappa measures and why it beats raw percent agreement
The observed minus chance over one minus chance intuition
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.