You are building an LLM judge for a customer service chatbot. The rubric rates 'helpfulness' on 1–5. Describe how you would add anchor examples to the judge prompt, explain the mechanism by which anchors reduce score variance, and estimate the expected impact.
Free · 2 AI evals / day
**Rubric anchoring** means attaching a concrete example response to each point on your scoring scale, instead of relying on adjectives like 'moderately helpful'. For a 1 to 5 helpfulness rubric, you write one short sample answer per level and label why it sits there.
For example: Score 1 is a flat 'I can't help with that' refusal. Score 3 partially answers but misses key details. Score 5 fully answers, anticipates the follow-up, and gives next steps. These examples live inside the judge prompt, before the candidate answer it has to score.
The reason this reduces variance is simple. Without anchors, the judge invents its own definition of 'what counts as a 3' on every run, and that definition wobbles. The same answer might get a 3 once and a 4 the next time. An anchor fixes the boundary between 3 and 4 to a shared concrete reference, so the judge interprets the scale the same way each time.
Anchors substantially reduce inter-run and inter-judge variance. The catch: if your anchors are badly chosen, they backfire by pulling every score toward the examples you showed.
**Rubric anchoring** means pairing each scale level with a concrete example response, not just a word like 'moderately helpful'. You place one labelled sample per level inside the judge prompt.
For 1 to 5 helpfulness: Score 1 is a flat refusal. Score 3 answers partially but misses key aspects. Score 5 answers fully, anticipates needs, and gives next steps. Each anchor carries a one-line justification.
The mechanism is boundary externalisation. Without anchors, the judge defines 'what is a 3' implicitly, and that definition drifts run to run. The same answer scores 3 then 4. An anchor pins the boundary to a shared reference. The judge stops re-deriving the scale and starts comparing against fixed exemplars.
Expected impact: substantially lower inter-run variance. For regression testing, a score shift below about 0.3 points is then more credibly model change, not judge drift.
The trap is anchoring bias. Anchors act as a cognitive prior. If they cluster in style or spacing, the judge compresses real answers toward them. Exemplar order also sways scores. So anchors must span the full range, stay diverse, and be validated against held-out human labels.
Quick reference
TL;DR
Per-level anchor examples pin the judge's implicit scale boundaries, substantially reducing inter-run variance. The catch: bad anchors induce anchoring bias, compressing scores toward the exemplars.
Imagine grading essays with a friend. If you only agree on words like 'a 3 is okay', you will drift apart, because 'okay' means different things on different days. Now you each tape one real sample essay next to each number on the scale. Suddenly you agree far more, because you are not guessing what a 3 feels like, you are comparing against a fixed example. That is what anchor examples do for an LLM judge: they replace a fuzzy mental boundary with a concrete reference. But there is a trap. If your three sample essays are all very similar, you start pulling every essay toward those samples, and the spread of your scores shrinks. The order you show them in can sway you too. So the samples must be chosen with care and shuffled.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks with cheap exact ground truth, like multiple choice or structured extraction Exact match against the gold answer is unbiased and free. A scored judge with anchors adds cost and bias for no gain.
Pairwise preference evaluation rather than scalar scoring Anchors target scale calibration, which pairwise comparison sidesteps. Pairwise instead fights position bias, a different problem.
Early exploratory eval where you have no validated human-labelled set yet Without held-out human scores you cannot tell calibration from compression, so anchors may quietly bias results you trust.
Highly diverse open-ended tasks where no small set of exemplars represents the space A handful of anchors cannot span the range, so they compress scores toward whatever narrow region they sample.
Quick scan · 2–4 min
Everything important, quickly.
6 min: anchor design per level, the boundary-externalisation mechanism, the substantial variance win, then the anchoring-bias backfire (compression, order effects) and the calibration fix against human labels.
Real-world examples
Real products, models, and research that use this idea.
1LangSmith and Braintrust let you embed per-level reference examples in judge prompt templates to stabilise scalar scoring.
2Prometheus 2, the dominant open-weight judge in 2026, is trained on rubric plus reference prompts that anchor each score band.
3RAGAS and DeepEval recommend few-shot scored exemplars in custom judge metrics, with a warning to span the full quality range.
4G-Eval style prompts use chain-of-thought plus rubric anchors, and teams report score compression when exemplars cluster.
5Anthropic Workbench judge templates surface few-shot exemplars and advise randomising their order to limit position effects.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect that your anchors are compressing scores rather than calibrating them?
Sign in or go Pro to see the approach for this follow-up.
QWhy does exemplar order in the judge prompt matter, and how do you neutralise it?
Sign in or go Pro to see the approach for this follow-up.
QHow do you choose how many anchors and where to place them on the scale?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Adding anchors and assuming variance is solved. Poorly spaced or same-style anchors induce anchoring bias, compressing scores toward the exemplars and creating false stability that hides real quality differences.