Human Eval Rater Agreement

You are building a golden dataset with three annotators. Someone warns you about inter-annotator agreement. What is it and what goes wrong if you skip measuring it?

Why do teams still pay for human evaluation when LLM-as-judge exists?

What does Cohen's kappa measure in human LLM evaluation?

When should you use Fleiss's kappa instead of Cohen's kappa in a human eval study?

How does Chatbot Arena compute ELO rankings, and what bias affects its validity?