What does Cohen's kappa measure in human LLM evaluation?

Attempt it

Cohen's kappa measures inter-rater agreement corrected for the agreement expected by chance. κ=0 means agreement no better than chance; κ=1 means perfect agreement. In LLM evaluation, κ>0.6 is considered 'substantial' agreement and κ>0.8 is 'almost perfect'. Values below 0.6 typically indicate the rubric is ambiguous or the task is inherently too subjective for reliable labeling.

Your answer

Quick reference

TL;DR

Cohen's kappa measures how much two human raters agree beyond chance. Above 0.6 is substantial, above 0.8 almost perfect; it sets the reliability ceiling any LLM judge can hope to match.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine two friends grading the same stack of essays as pass or fail. If they agree on 80 out of 100, that sounds great. But suppose almost every essay is a clear pass, so even two people guessing 'pass' every time would agree most of the time. Raw agreement is fooled by easy cases. Kappa is a smarter score: it asks how much the two friends agree above what pure luck would give. A kappa near zero means they were basically guessing in lockstep. A kappa near one means they truly see eye to eye on the hard calls too. In LLM work, you first check that humans agree with each other before you trust any automated grader to match them.

Key concepts

Human rater agreement Cohen's kappa Inter-rater reliability LLM-as-judge

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: what kappa measures, the observed minus chance intuition, the agreement bands, the Cohen vs Fleiss vs weighted variants, and why human agreement is the ceiling for any judge.

Real-world examples

Real products, models, and research that use this idea.

Anthropic and OpenAI both double-label safety eval sets and report inter-annotator kappa before treating labels as ground truth.
RAGAS and LangSmith documentation recommend validating an LLM judge against a human-labelled holdout using kappa or correlation.
MT-Bench and Chatbot Arena studies report human agreement levels to justify using LLM judges as a proxy.
Scale AI and Surge AI annotation pipelines surface per-task inter-rater agreement as a first-class quality signal.
Prometheus 2 evaluations report agreement with human labels to show the open-weight judge tracks expert consensus.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can two raters show 90 percent agreement yet a kappa near zero?

Sign in or go Pro to see the approach for this follow-up.

QHow do you decide whether to use Cohen's, Fleiss's, or weighted kappa?

Sign in or go Pro to see the approach for this follow-up.

QHow does human kappa bound what an LLM judge can achieve?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reporting raw percent agreement instead of kappa. High agreement on a skewed label set can look strong while raters are effectively guessing in lockstep with the majority class.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.