When should you use Fleiss's kappa instead of Cohen's kappa in a human eval study?

Attempt it

Quick reference

TL;DR

Use Fleiss's kappa when three or more raters label the same items. Cohen's kappa is defined only for exactly two raters, so it cannot scale to a judge panel or a multi-annotator pool.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading the same stack of essays. With two teachers, you line up their grades side by side and count how often they match beyond what luck alone would give. That matched beyond luck score is Cohen's kappa, and it only works for a pair. Now bring in five teachers per essay. There is no neat pair to compare anymore, because every essay carries five opinions at once. Fleiss's kappa solves this by asking a different question: for each essay, how often did any two of the five graders happen to agree? It averages that across all essays and corrects for chance. So Cohen's is for couples, Fleiss's is for the whole committee. You pick based on how many raters touch each item, not on the label scale or who the raters happen to be.

Key concepts

Inter-rater agreement Fleiss's kappa Cohen's kappa Human eval rater agreement

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: the exactly two raters limit of Cohen's, Fleiss's as the multi-rater generalization, the shared chance-corrected formula, fixed versus interchangeable raters, and ordinal-scale alternatives.

Real-world examples

Real products, models, and research that use this idea.

MT-Bench and Chatbot Arena human studies report inter-rater agreement so judge-model rankings can be validated against people.
LangSmith and Braintrust let teams collect multi-annotator labels on a holdout set to calibrate LLM judges against human consensus.
RAGAS faithfulness validation uses small human-labeled panels where three-plus annotators per item make Fleiss's kappa the right agreement metric.
Scale AI and Surge AI annotation pipelines compute multi-rater agreement to flag low-quality or systematically biased crowd workers.
Anthropic and OpenAI preference-data pipelines track annotator agreement to gate which labels enter reward-model training.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can averaging pairwise Cohen's kappa across a panel give a different answer than Fleiss's kappa?

Sign in or go Pro to see the approach for this follow-up.

QYour annotators use a 1-to-5 ordinal quality scale. Which agreement metric do you reach for and why?

Sign in or go Pro to see the approach for this follow-up.

QAgreement looks high by eye but Fleiss's kappa is near zero. What is happening?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reaching for Cohen's kappa with a 3-judge panel by averaging pairwise scores. That hides a systematic rater and is not what Cohen's measures. Use Fleiss's kappa, which is built for three or more raters.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.