Design an LLM judge prompt with anchor examples and explain how they reduce score variance

**Rubric anchoring** means attaching a concrete example response to each point on your scoring scale, instead of relying on adjectives like 'moderately helpful'. For a 1 to 5 helpfulness rubric, you write one short sample answer per level and label why it sits there. For example: Score 1 is a flat 'I can't help with that' refusal. Score 3 partially answers but misses key details. Score 5 fully answers, anticipates the follow-up, and gives next steps. These examples live inside the judge prompt, before the candidate answer it has to score. The reason this reduces variance is simple. Without anchors, the judge invents its own definition of 'what counts as a 3' on every run, and that definition wobbles. The same answer might get a 3 once and a 4 the next time. An anchor fixes the boundary between 3 and 4 to a shared concrete reference, so the judge interprets the scale the same way each time. Anchors substantially reduce inter-run and inter-judge variance. The catch: if your anchors are badly chosen, they backfire by pulling every score toward the examples you showed.

**Rubric anchoring** means pairing each scale level with a concrete example response, not just a word like 'moderately helpful'. You place one labelled sample per level inside the judge prompt. For 1 to 5 helpfulness: Score 1 is a flat refusal. Score 3 answers partially but misses key aspects. Score 5 answers fully, anticipates needs, and gives next steps. Each anchor carries a one-line justification. The mechanism is boundary externalisation. Without anchors, the judge defines 'what is a 3' implicitly, and that definition drifts run to run. The same answer scores 3 then 4. An anchor pins the boundary to a shared reference. The judge stops re-deriving the scale and starts comparing against fixed exemplars. Expected impact: substantially lower inter-run variance. For regression testing, a score shift below about 0.3 points is then more credibly model change, not judge drift. The trap is anchoring bias. Anchors act as a cognitive prior. If they cluster in style or spacing, the judge compresses real answers toward them. Exemplar order also sways scores. So anchors must span the full range, stay diverse, and be validated against held-out human labels.