What does Cohen's kappa measure in human LLM evaluation?

**Cohen's kappa** measures how much two human raters agree when they label the same items, after subtracting the agreement you would expect from pure chance. It answers a sharper question than raw percent agreement: are these raters genuinely consistent, or do they just happen to pick the same common answer a lot? The scale runs from 0 to 1, and can dip negative. A kappa of 0 means agreement is no better than chance. A kappa of 1 means perfect agreement on every item. For LLM evaluation, the common bands are simple. Above 0.6 is substantial, and above 0.8 is almost perfect. Below 0.6 usually means your rubric is ambiguous or the task is too subjective for reliable labels. Why do we care in LLM work? Before you trust an automated grader (an LLM-as-judge), you check that humans agree with each other first. Human agreement sets the ceiling. If two experts only reach kappa 0.5, no judge can reliably do better, because there is no stable ground truth to match.

**Cohen's kappa** measures inter-rater agreement between two human annotators, corrected for chance agreement. It is the standard reliability check before you trust any label set or automated judge. Raw percent agreement is misleading on skewed data. If 90 percent of items are 'good', two raters guessing 'good' agree most of the time. Kappa subtracts that expected luck. Kappa is: $$\kappa = \frac{p_o - p_e}{1 - p_e}$$ Observed agreement is `p_o` and chance agreement is `p_e`. Zero means chance level; one means perfect. The accepted bands are easy to recall. Above 0.6 is substantial, above 0.8 is almost perfect. Below 0.6 flags an ambiguous rubric or a task too subjective to label reliably. The key role in evaluation is the ceiling. Human agreement bounds what any LLM judge can achieve. If your experts reach only kappa 0.55, a judge cannot reliably beat them, because the ground truth itself is noisy. So you measure human kappa first, then ask whether the judge agrees with humans at a comparable level.