Cohens Kappa

You are building a golden dataset with three annotators. Someone warns you about inter-annotator agreement. What is it and what goes wrong if you skip measuring it?

What does Cohen's kappa measure in human LLM evaluation?

How do you measure whether an LLM judge is well-calibrated against human raters?

Complete: Cohen's kappa >___ is 'substantial' agreement; >___ is 'almost perfect' agreement