Use Fleiss's kappa when three or more raters label the same items. Cohen's kappa is defined only for exactly two raters, so it cannot scale to a judge panel or a multi-annotator pool.
Imagine grading the same stack of essays. With two teachers, you line up their grades side by side and count how often they match beyond what luck alone would give. That matched beyond luck score is Cohen's kappa, and it only works for a pair. Now bring in five teachers per essay. There is no neat pair to compare anymore, because every essay carries five opinions at once. Fleiss's kappa solves this by asking a different question: for each essay, how often did any two of the five graders happen to agree? It averages that across all essays and corrects for chance. So Cohen's is for couples, Fleiss's is for the whole committee. You pick based on how many raters touch each item, not on the label scale or who the raters happen to be.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Inter-rater agreement is the backbone of any credible human eval study. When you collect human labels to validate an LLM judge, or to build a preference dataset, you need a number that says how much your annotators actually agree once you strip out the agreement you would get from chance alone. Raw percent agreement is not that number, because two random labelers still agree often on a coarse scale. On a binary task, two coin-flippers match roughly half the time while learning nothing, so a metric that reports fifty percent agreement as if it were a real signal is actively misleading.
The kappa family supplies the chance correction. It rescales raw agreement against the agreement you would expect from raters guessing in proportion to the label frequencies, so that a kappa of zero means no skill and a kappa of one means perfect concordance. The question here is narrow but important: Cohen's kappa and Fleiss's kappa are not interchangeable, and the choice between them is dictated by one structural fact about your data, the number of raters who label each item.
This deep dive walks the two estimators, the shared formula, the subtle assumption differences about whether raters are fixed or interchangeable, the interpretation thresholds and the famous paradox that distorts them, and the cases where you should abandon both kappas for a more general metric. The thread running through all of it is that the metric choice is mechanical once you describe your data correctly.
Cohen's kappa: the two-rater measure
Cohen's kappa measures chance-corrected agreement between exactly two raters who each label the same set of items. You cross-tabulate rater A against rater B into a contingency table, read off the observed agreement along the diagonal, and compute the expected agreement from each rater's own label distribution. The expected term multiplies rater A's marginal probability of each label by rater B's marginal probability of the same label and sums across labels, capturing how often the two would coincide purely by independent chance.
The defining constraint is the count: two raters, both labeling every item. The contingency table is inherently a two-way object, one rater per axis. There is no clean way to put three raters into a single such table, which is exactly why Cohen's stops at a pair. You could build separate tables for every pair, but that is no longer one statistic and, as we will see, averaging those pairwise values does not reconstruct the true panel agreement.
A second point that trips up candidates: Cohen's is not limited to binary labels. It works on any nominal scale with any number of categories, and the weighted variant extends it to ordinal scales by penalizing far disagreements more than near ones, using a weight matrix over the label distances. The binary-only claim in the question is a distractor that confuses the simplest illustrative case with the actual definition. Likewise, Cohen's makes no assumption about who the raters are demographically; it only cares about their label distributions, so the demographic-homogeneity option is also a distractor.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- MT-Bench and Chatbot Arena human studies report inter-rater agreement so judge-model rankings can be validated against people.
- LangSmith and Braintrust let teams collect multi-annotator labels on a holdout set to calibrate LLM judges against human consensus.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can averaging pairwise Cohen's kappa across a panel give a different answer than Fleiss's kappa?
Cohen's uses each pair's own marginals for chance, while Fleiss's pools marginals across all raters. Averaging pairs also weights differently and can hide one systematically biased rater that Fleiss's would dilute.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for Cohen's kappa with a 3-judge panel by averaging pairwise scores. That hides a systematic rater and is not what Cohen's measures. Use Fleiss's kappa, which is built for three or more raters.
60 second bullets to scan on the way to the call.
The exactly two raters constraint on Cohen's kappa
Fleiss's kappa as the generalization to three or more raters
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.