You are building a golden dataset with three annotators. Someone warns you about inter annotator agreement. What is it and what goes wrong if you skip measuring it?

Flashcard·Easy·4.0 · 0·~30s·Asked atGraphcore Linkedin Stability Ai·Relevant atAnthropic

Attempt it

Inter-annotator agreement (IAA) measures how consistently multiple human raters label the same items. If you skip measuring it, you cannot distinguish between 'the model is bad' and 'the raters disagree on what good looks like.' Low IAA means your rubric is ambiguous or the task is inherently subjective, and any eval score built on those labels is unreliable. The standard metric is Cohen's kappa (for two raters) or Fleiss' kappa (for three or more), where values above 0.7 generally indicate reliable labels.

Your answer

**Inter-annotator agreement (IAA)** measures how often multiple human raters assign the same label to the same item. If you have three annotators rating model outputs as 'good' or 'bad,' IAA tells you how consistently they agree. Why does this matter? Because if your annotators disagree frequently, the labels in your golden dataset are noisy. A model evaluated against noisy labels gets unreliable scores. You cannot tell whether a low score means the model is bad or the annotators could not agree on what good looks like. The standard metric is Cohen's kappa for two raters or Fleiss' kappa for three or more. Kappa improves on raw percentage agreement by accounting for chance: if 90% of outputs are 'good,' two raters randomly labeling everything 'good' would agree 81% of the time by chance alone. Kappa subtracts this expected agreement and tells you how much agreement exists above chance. General thresholds: kappa above 0.8 is strong agreement. Kappa between 0.6 and 0.8 is moderate. Below 0.6, the labels are too noisy for reliable evaluation. When kappa is low, the fix is usually the rubric: the instructions are ambiguous, the categories are poorly defined, or the task is inherently subjective and needs a different evaluation approach.

**Inter-annotator agreement (IAA)** measures how consistently multiple human raters label the same items. It is the diagnostic that tells you whether your evaluation labels are reliable enough to trust. The measurement matters for a specific reason: it separates model quality from labeling quality. If your annotators disagree on 40% of items, an eval score built on those labels has 40% noise. A model that looks like it improved might just have gotten lucky on the contested items. A model that looks like it regressed might be fine on the items the annotators agree about. The standard metrics are Cohen's kappa (two raters) and Fleiss' kappa (three or more). Kappa accounts for chance agreement, which raw percentage agreement does not. If 95% of outputs are labeled 'good,' two raters both labeling everything 'good' agree 90% by chance. Raw agreement says 90%, which sounds excellent. Kappa says near zero, which correctly reflects that neither rater is making meaningful distinctions. Thresholds vary by domain, but a common guideline: kappa above 0.8 indicates strong agreement suitable for evaluation. Between 0.6 and 0.8 is acceptable for many tasks. Below 0.6, the rubric needs revision before the labels can be used. If you skip measuring IAA, you lose the ability to diagnose evaluation failures. When scores are bad, you do not know if the model failed or the labels are unreliable. When scores are good, you do not know if the model succeeded or the noisy labels happened to align favorably.

TL;DR

Inter-annotator agreement measures how consistently multiple raters label the same items, revealing whether your evaluation labels are reliable enough to trust.

Memory aid

Easy to grasp

Imagine three teachers grading the same stack of essays. If they all give similar grades, you can trust the grades to mean something. But if teacher A gives a B+ while teacher B gives a D and teacher C gives an A, the grades are useless because the teachers do not agree on what good writing looks like. Inter-annotator agreement is the number that tells you whether your graders are on the same page. You measure it before using the grades for anything important. If the number is low, you fix the grading rubric (make the instructions clearer) and try again. If you skip this check, you might blame the student for bad writing when the real problem is that the teachers cannot agree.

Key concepts

Inter-Annotator Agreement Cohen's Kappa Label Reliability Rubric Ambiguity

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

When you build a golden dataset, the quality of the labels determines the quality of every evaluation that uses them. Inter-annotator agreement is the measurement that tells you whether your labels are trustworthy.

This walkthrough covers what IAA is, why it matters, how to measure it, and what to do when it is low.

What IAA measures and why it matters

Inter-annotator agreement quantifies how often multiple raters assign the same label to the same item. If three annotators rate 100 model outputs and agree on 90 of them, agreement is high. If they agree on only 55, something is wrong with the labeling process.

The reason IAA matters is that evaluation scores are only as reliable as the labels they are computed against. If your golden dataset has noisy labels (annotators disagreed but you picked one label arbitrarily), the model's score includes that noise. A 5% improvement in model quality might be invisible in a dataset with 20% label noise. Conversely, a 3% regression might appear as an improvement if the noisy labels happened to align favorably.

Measuring IAA before using labels for evaluation lets you diagnose and fix reliability problems at the source. It is cheaper to fix a rubric than to debug a mysterious eval score six months later.

Why kappa beats raw percentage agreement

Three causes of low IAA and how to fix each

The pilot then scale workflow

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Scale AI measures inter-annotator agreement as a standard quality metric for every annotation project, flagging batches where kappa drops below threshold for rubric revision.
Chatbot Arena sidesteps the IAA problem by collecting pairwise preferences from thousands of users, where the volume compensates for individual disagreement.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the difference between Cohen's kappa and Fleiss' kappa, and when do you use each?

Cohen's kappa works for exactly two raters on the same items. Fleiss' kappa generalizes to three or more raters, allowing different subsets of raters to label different items. Use Cohen's when you have a fixed pair of annotators; use Fleiss' when you have a pool of annotators and each item is rated by a subset.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using raw percentage agreement instead of kappa. Two raters can agree 90% of the time by chance alone if one class dominates, making raw agreement misleadingly high.

60 second bullets to scan on the way to the call.

Define inter-annotator agreement as a measure of label consistency across raters
Explain why skipping IAA measurement makes evaluation scores uninterpretable

Primary sources. Browse if you want the original framing.