You are building an LLM judge for a customer service chatbot. The rubric rates 'helpfulness' on 1–5. Describe how you would add anchor examples to the judge prompt, explain the mechanism by which anchors reduce score variance, and estimate the expected impact.
Per-level anchor examples pin the judge's implicit scale boundaries, substantially reducing inter-run variance. The catch: bad anchors induce anchoring bias, compressing scores toward the exemplars.
Imagine grading essays with a friend. If you only agree on words like 'a 3 is okay', you will drift apart, because 'okay' means different things on different days. Now you each tape one real sample essay next to each number on the scale. Suddenly you agree far more, because you are not guessing what a 3 feels like, you are comparing against a fixed example. That is what anchor examples do for an LLM judge: they replace a fuzzy mental boundary with a concrete reference. But there is a trap. If your three sample essays are all very similar, you start pulling every essay toward those samples, and the spread of your scores shrinks. The order you show them in can sway you too. So the samples must be chosen with care and shuffled.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Rubric anchoring is the practice of pairing each point on a scoring scale with a concrete example response, so the LLM judge scores by comparison against fixed references rather than by re-interpreting adjectives like 'moderately helpful' on every call. It is one of the highest-leverage moves in judge prompt design, and also one of the easiest to get subtly wrong.
This question asks you to do three things: design the anchors, explain why they cut score variance, and estimate the impact. A senior answer does all three and then adds the part that separates a calibration expert from a prompt tinkerer: anchors are themselves a bias source. They can compress the score distribution toward the exemplars and pick up exemplar-order effects. The deep dive walks the design, the variance mechanism, the expected magnitude, the anchoring-bias backfire, and the calibration discipline that keeps anchors honest.
Designing the anchors: one concrete exemplar per level
For a 1 to 5 helpfulness rubric on a customer service chatbot, write one short example response per level, each with a label and a one-line justification, and place them in the judge prompt before the candidate answer.
A workable set: Score 1 is a flat refusal, 'I can't help with that', with no attempt to assist. Score 2 offers tangentially related information that does not address the question. Score 3 answers partially with correct information but misses key aspects. Score 4 fully and accurately answers with no missing critical detail. Score 5 answers comprehensively, anticipates the follow-up need, and provides actionable next steps.
The justification line is not decoration. It tells the judge which property moved the example up or down a level, so the judge keys on the underlying quality dimension rather than on surface length or tone. Anchor the extremes and the contested boundaries first, because those are where disagreement concentrates.
There is a format choice underneath the design. You can inline the anchors as a static block in the system prompt, which is cheapest and cacheable, or retrieve the nearest scored exemplars per candidate, which is sharper but adds a retrieval step and a fresh chance to leak label information. For a fixed-scale helpfulness rubric the static block is almost always right: five anchors, one per level, frozen and versioned alongside the rubric text. Keep each anchor short. A paragraph-long exemplar wastes context budget and invites the judge to imitate its phrasing rather than its quality level.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith and Braintrust let you embed per-level reference examples in judge prompt templates to stabilise scalar scoring.
- Prometheus 2, the dominant open-weight judge in 2026, is trained on rubric plus reference prompts that anchor each score band.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect that your anchors are compressing scores rather than calibrating them?
Plot the judge score histogram before and after adding anchors. A collapse toward the anchored region, especially loss of tail mass at 1 and 5, signals compression. Cross-check against a human-scored holdout: variance should fall while rank correlation holds or rises, not falls.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Adding anchors and assuming variance is solved. Poorly spaced or same-style anchors induce anchoring bias, compressing scores toward the exemplars and creating false stability that hides real quality differences.
60 second bullets to scan on the way to the call.
What a per-level anchor example looks like for a 1 to 5 rubric
The boundary externalisation mechanism behind variance reduction
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.