Why do anchor examples in an LLM judge prompt dramatically reduce score variance?

An **LLM judge** is a language model you ask to score another model's output, say 'rate this answer's helpfulness from 1 to 5'. The problem with a bare scale is that the model has no shared definition of what each number means. Without examples, the judge invents its own interpretation each time. On one run a thorough but wordy answer is a 4; on the next run the same answer is a 3, because the judge's internal sense of the boundary drifted. Two different judges disagree even more. Anchor examples fix this. You include one concrete worked example for each score level inside the prompt: here is what a 2 looks like, here is a 3, here is a 5. Now the judge calibrates against shared references instead of guessing. This substantially reduces variance across runs and across judges. The wrong options miss the mechanism. It is not that longer prompts make the judge think harder (B), not that the judge copies a template (C), and not recency exploitation (D). The effect is calibration: anchors define the scale so everyone reads it the same way.

The correct answer is A. **Anchor examples** work by calibration, not by prompt length or templating. A bare rubric like 'rate helpfulness 1 to 5' under-specifies the scale. The judge fills the gap with its own implicit boundaries. Those boundaries shift across runs, across phrasings, and across judge models. The result is high variance with no stable meaning behind a 3 versus a 4. Anchor examples attach a concrete reference to each level. The judge no longer guesses where a boundary sits. It matches the candidate against worked examples. Inter-run and inter-judge variance drops substantially. The distractors name real phenomena but wrong causes. Longer prompts do not reliably help. Template copying is not the mechanism. Recency bias is an artifact to suppress, not a feature. The senior caveat is that anchors which reduce variance can still shift the mean. If every anchor is a strong answer, the judge calibrates upward and inflates scores. Use balanced anchors that span the full quality range. Pair anchoring with blind scoring so the judge cannot see which system produced the output.