Complete: Cohen's kappa >___ is 'substantial' agreement; >___ is 'almost perfect' agreement

When you replace a human grader with an **LLM judge**, you need proof the judge agrees with humans. The natural instinct is to report percent agreement: out of 200 examples, the judge and the human gave the same label on 180, so 90 percent. The problem is that raw agreement is inflated whenever one label dominates. Cohen's kappa fixes this. It measures agreement above what you would expect by chance, so a lucky-looking 90 percent on a skewed set can collapse to a kappa near zero. Kappa runs from -1 to 1, where 0 means chance-level agreement and 1 means perfect agreement. The community uses named bands from a 1977 paper by Landis and Koch. Kappa above 0.6 is called substantial agreement and is the usual minimum for trusting a judge in production. Kappa above 0.8 is almost perfect, the gold standard where the judge is effectively as good as a second human. Below 0.6, the judge disagrees with humans too often to ship as a sole signal. You would either fix the rubric, swap the judge model, or keep a human in the loop.

**Cohen's kappa** measures how much an LLM judge and human raters agree beyond chance. Raw percent agreement is misleading on skewed label sets, so kappa is the standard calibration metric. The formula subtracts chance agreement from observed agreement and normalises by the headroom above chance. Kappa is: $\kappa = \frac{p_o - p_e}{1 - p_e}$ Here, observed agreement `p_o` is how often judge and human labelled an item identically. Chance agreement `p_e` comes from each rater's marginal label frequencies. The interpretation bands come from Landis and Koch. Above 0.6 is substantial agreement, the usual minimum bar for a production judge. Above 0.8 is almost perfect. Between 0.4 and 0.6 is moderate, often fine for a coarse pre-filter but not for a release gate. Below 0.6 the judge is too unreliable to trust alone. You diagnose by class, since one rubric dimension can drag kappa down. The fix is usually a clearer rubric, a stronger judge model, or anchor examples. Recalibrate quarterly, because model versions drift.