You are building a golden dataset with three annotators. Someone warns you about inter-annotator agreement. What is it and what goes wrong if you skip measuring it?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Inter-annotator agreement (IAA) measures how consistently multiple human raters label the same items. If you skip measuring it, you cannot distinguish between 'the model is bad' and 'the raters disagree on what good looks like.' Low IAA means your rubric is ambiguous or the task is inherently subjective, and any eval score built on those labels is unreliable. The standard metric is Cohen's kappa (for two raters) or Fleiss' kappa (for three or more), where values above 0.7 generally indicate reliable labels.

Your answer

**Inter-annotator agreement (IAA)** measures how often multiple human raters assign the same label to the same item. If you have three annotators rating model outputs as 'good' or 'bad,' IAA tells you how consistently they agree. Why does this matter? Because if your annotators disagree frequently, the labels in your golden dataset are noisy. A model evaluated against noisy labels gets unreliable scores. You cannot tell whether a low score means the model is bad or the annotators could not agree on what good looks like. The standard metric is Cohen's kappa for two raters or Fleiss' kappa for three or more. Kappa improves on raw percentage agreement by accounting for chance: if 90% of outputs are 'good,' two raters randomly labeling everything 'good' would agree 81% of the time by chance alone. Kappa subtracts this expected agreement and tells you how much agreement exists above chance. General thresholds: kappa above 0.8 is strong agreement. Kappa between 0.6 and 0.8 is moderate. Below 0.6, the labels are too noisy for reliable evaluation. When kappa is low, the fix is usually the rubric: the instructions are ambiguous, the categories are poorly defined, or the task is inherently subjective and needs a different evaluation approach.

**Inter-annotator agreement (IAA)** measures how consistently multiple human raters label the same items. It is the diagnostic that tells you whether your evaluation labels are reliable enough to trust. The measurement matters for a specific reason: it separates model quality from labeling quality. If your annotators disagree on 40% of items, an eval score built on those labels has 40% noise. A model that looks like it improved might just have gotten lucky on the contested items. A model that looks like it regressed might be fine on the items the annotators agree about. The standard metrics are Cohen's kappa (two raters) and Fleiss' kappa (three or more). Kappa accounts for chance agreement, which raw percentage agreement does not. If 95% of outputs are labeled 'good,' two raters both labeling everything 'good' agree 90% by chance. Raw agreement says 90%, which sounds excellent. Kappa says near zero, which correctly reflects that neither rater is making meaningful distinctions. Thresholds vary by domain, but a common guideline: kappa above 0.8 indicates strong agreement suitable for evaluation. Between 0.6 and 0.8 is acceptable for many tasks. Below 0.6, the rubric needs revision before the labels can be used. If you skip measuring IAA, you lose the ability to diagnose evaluation failures. When scores are bad, you do not know if the model failed or the labels are unreliable. When scores are good, you do not know if the model succeeded or the noisy labels happened to align favorably.

Quick reference

TL;DR

Inter-annotator agreement measures how consistently multiple raters label the same items, revealing whether your evaluation labels are reliable enough to trust.

Memory aid

Easy to grasp

Imagine three teachers grading the same stack of essays. If they all give similar grades, you can trust the grades to mean something. But if teacher A gives a B+ while teacher B gives a D and teacher C gives an A, the grades are useless because the teachers do not agree on what good writing looks like. Inter-annotator agreement is the number that tells you whether your graders are on the same page. You measure it before using the grades for anything important. If the number is low, you fix the grading rubric (make the instructions clearer) and try again. If you skip this check, you might blame the student for bad writing when the real problem is that the teachers cannot agree.

Key concepts

Inter-Annotator Agreement Cohen's Kappa Label Reliability Rubric Ambiguity

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define IAA, explain why it separates model quality from labeling quality, name kappa as the standard metric with chance correction, state thresholds, walk through the three causes of low IAA, and describe the pilot then scale workflow.

Real-world examples

Real products, models, and research that use this idea.

Scale AI measures inter-annotator agreement as a standard quality metric for every annotation project, flagging batches where kappa drops below threshold for rubric revision.
Chatbot Arena sidesteps the IAA problem by collecting pairwise preferences from thousands of users, where the volume compensates for individual disagreement.
Academic NLP datasets like SQuAD report inter-annotator agreement alongside performance numbers so that readers can judge whether the benchmark's ceiling is limited by annotation noise.
Teams building evaluation datasets for Ragas and DeepEval measure IAA on a pilot batch before scaling annotation, treating low kappa as a blocker.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the difference between Cohen's kappa and Fleiss' kappa, and when do you use each?

QWhen IAA is low because the task is genuinely subjective, how do you evaluate reliably?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Using raw percentage agreement instead of kappa. Two raters can agree 90% of the time by chance alone if one class dominates, making raw agreement misleadingly high.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.