Walk through how to calibrate an LLM-as-judge against human labels with Cohen's kappa

Short answer·Hard·4.0 · 0·~3 min·Asked atFiddler Ai Servicenow Uniphore

Attempt it

Before trusting a new LLM-as-judge prompt in production, you want to calibrate it against a small human labeled set. Walk through the workflow, what statistic you compute, and what threshold tells you the judge is good enough to ship.

Free · 2 AI evals / day

An LLM-as-judge is a model you use to score other model outputs (for example, is this answer factually correct, is this summary faithful). Before you trust it in production, you need to prove it agrees with humans on a small set. The workflow has five steps. First, sample 100 to 300 traces stratified across the dimensions that matter (task type, model used, easy vs hard examples) so you cover the space evenly. Second, have two or three humans label each example using the same rubric the judge will use. Third, reconcile disagreements: the humans discuss and pick one label per item, giving you a consensus set. Fourth, run the judge on the same set and compute Cohen's kappa between the judge and the consensus. Fifth, compare kappa to a threshold: roughly 0.6 is the typical ship bar. Kappa above 0.6 is substantial agreement, 0.4 to 0.6 is moderate (iterate the prompt), and below 0.4 is unusable. Always also compute inter-human kappa as the ceiling. After shipping, re-run the calibration on a rolling sample to catch drift.

Calibrating an LLM-as-judge against humans is a five-step process, and each step has a failure mode. **Sample.** Pull 100 to 300 traces from production, stratified across the dimensions you actually care about: task type, model version, expected difficulty, customer segment. A random sample over the whole population under-represents rare cases; stratification fixes that. **Label.** Two or three humans label every item against the same rubric the judge will use. The rubric must be the *identical* rubric, not a paraphrase, otherwise you are measuring rubric drift, not judge quality. **Reconcile.** Where humans disagree, the labelers talk it out and pick one consensus label. Items where consensus cannot be reached are dropped or flagged as ambiguous; do not let majority vote pretend to be consensus on a 2-vs-1 split. **Compute kappa.** Run the judge on the reconciled set and compute Cohen's kappa between judge labels and consensus labels. Cohen's kappa corrects for agreement by chance, which raw accuracy does not. Use weighted kappa for ordinal rubrics (1-5 quality scores), unweighted for binary or nominal labels. **Set the threshold.** Substantial agreement is kappa > 0.6, and that is the typical ship bar. Compute inter-human kappa on the same set as a ceiling: if humans only agree at 0.55, the judge cannot exceed that on a fair test, and you should suspect the rubric is the problem. After shipping, run the calibration on a rolling sample every few weeks to catch drift from model updates or prompt edits.

TL;DR

Stratify-sample 100-300 traces, get multi-human consensus labels, compute Cohen's kappa between the judge and the consensus, ship if kappa exceeds 0.6 and stays under the inter-human ceiling.

Memory aid

Easy to grasp

Imagine you hired a robot referee to call balls and strikes in a baseball game, and you want to know whether to trust it. You cannot just ask if it sounds confident. You pick a few hundred pitches that cover the easy ones and the close ones, and you have three umpires call each one. When the umpires disagree, they talk it out and pick one answer. Now you have a gold-standard label for each pitch. You let the robot call those same pitches and you count: out of all the calls, how often does the robot match the umpires more than you would expect from luck? That number is called kappa. If it is above about 0.6, the robot is good enough to ship. If the umpires themselves only agree 0.55 of the time, no robot can do better than that on a fair test. You keep checking the robot every few weeks so it does not drift.

Key concepts

Observability and tracing LLM evaluation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

An LLM-as-judge is the workhorse of scaled evaluation: cheaper than humans, faster than humans, and the only way to score the millions of production traces that flow through a modern LLM system. The cost of speed is that an uncalibrated judge can be confidently wrong in ways that humans never would be, and that confidence is invisible in production.

Calibration is the practice of measuring how well the judge agrees with humans on a small reference set before trusting it at scale. This deep dive covers the five steps of the calibration loop, why Cohen's kappa is the right statistic, the inter-human ceiling, and how to keep the judge calibrated as models and prompts drift over time.

Sampling: stratification is more important than size

The first instinct on a calibration set is to grab 500 random traces and label them. This is usually wrong. Production traffic is heavily skewed: 80 to 95 percent of traces are easy cases on which any judge will score well, and the calibration kappa is dominated by those easy items even though the failure cases are what you actually need to measure.

Stratification dimensions

The right move is to define stratification dimensions up front: task type (summarization, classification, agent step), model used, expected difficulty (cheap heuristics like length, entropy, retrieval quality), and customer segment if that matters. Pull a fixed number of items per cell, oversampling the long tail.

A 200-item stratified set typically gives a more reliable kappa than a 500-item random set, because the rare cases are represented well enough for the statistic to be sensitive to them. Confidence intervals on kappa widen fast below 100 items; 100 is the practical floor, 200 to 300 the comfortable range.

Labeling and reconciliation: humans are the substrate

Cohen's kappa: what it corrects for and how to use it

The drift loop: calibration is not a one-time event

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Cohen's kappa value	Interpretation	Action
< 0.0	Worse than chance	Judge is broken; rewrite the prompt
0.0 to 0.4	Slight to fair	Unusable; iterate the rubric or judge prompt
0.4 to 0.6	Moderate	Iterate; do not ship yet
0.6 to 0.8	Substantial	Ship bar; verify by slice
0.8 to 1.0	Almost perfect	Excellent, but check if humans agree this much (suspicious if rubric is hard)

Real products, models, and research that use this idea.

Langfuse, Braintrust, and Arize Phoenix all expose human-labeling UIs that emit labels in a shape suitable for kappa computation against judge scores.
OpenAI's published evals work and the Anthropic capabilities-eval pipelines explicitly track inter-annotator agreement (often kappa or alpha) before reporting judge results.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the difference between Cohen's kappa and Krippendorff's alpha, and when should you use each?

Cohen's kappa is for two raters and nominal or ordinal labels with simple weights. Krippendorff's alpha generalizes to more than two raters and missing data. Once you reconcile to a consensus, two-rater kappa is usually what you want.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Shipping a judge after eyeballing a few examples instead of measuring kappa against multi-human consensus on a stratified sample, then being surprised when the judge silently drifts in production.

60 second bullets to scan on the way to the call.

Why stratified sampling beats random sampling for calibration sets
The role of multi-human labeling and consensus reconciliation

Primary sources. Browse if you want the original framing.