Anchor examples pin each score level to a concrete reference, so the judge stops interpreting the scale idiosyncratically. That alignment substantially reduces inter-run and inter-judge variance.
Imagine grading essays without a rubric. One teacher's 4 out of 5 is another teacher's 3, because nobody agreed on what each number means. Now imagine each teacher is handed a sample essay already marked 3, another marked 4, and another marked 5. Suddenly everyone grades the same essay almost the same way, because they all calibrate against the same examples. Anchor examples do exactly that for an LLM judge: one worked example per score level tells the model concretely what a 3 looks like versus a 4. The judge stops guessing where the boundaries sit and starts matching against shared references. The catch: if your anchors are lopsided, say only high-quality examples, you nudge the judge to score everything high. Balanced anchors fix the variance without quietly biasing the average.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Anchor examples in an LLM judge prompt are worked exemplars, one per score level, that show the judge concretely what a 2, a 3, or a 5 looks like. They are the single highest-leverage move for reducing judge variance, and the reason is calibration, not any of the surface explanations the distractors offer.
The deeper framing is that an LLM-as-judge is a measurement instrument, and like any instrument it has both variance (noise, the spread of repeated readings) and bias (systematic offset from the true value). Anchoring is fundamentally a variance-reduction technique. The subtle and interview-worthy point is that it is not free: a careless anchor set can lower variance while introducing bias, which is exactly the kind of trade that quietly corrupts an eval dashboard.
This deep dive walks the mechanism behind the variance reduction, the empirical magnitude, why the three distractors misattribute the cause, and the senior-level trap that anchoring introduces. It then covers the mitigation stack (balanced anchors, blind scoring, order randomization) and how to validate that anchoring actually helped rather than hurt.
Why a bare rubric drifts
A prompt that says 'rate helpfulness from 1 to 5' specifies the range but not the meaning. The judge must decide, on its own, where the boundary between a 3 and a 4 sits. That decision is a latent interpretation drawn from the model's priors, and it is not stable.
The instability has three sources, all of which matter in practice. Across repeated runs at any temperature above zero, sampling alone perturbs the score. Across prompt edits, even a reworded rubric line shifts how the judge reads the scale. Across judge models or model versions, the implicit boundary can move substantially, because each model learned a different distribution of what 'a 4 out of 5 answer' looks like.
The same candidate answer scored on two runs can therefore land on different numbers because the implicit boundary moved, not because the answer changed. None of this reflects a real change in quality; it is pure measurement noise from an under-specified scale. The human-annotation literature calls this label-interpretation variance, and it solves the human version with calibration sessions, where raters score gold examples together before labeling independently.
This is the core failure: the scale has no shared definition, so every score is anchored to a different mental ruler. High variance is the symptom, and it makes rankings unreliable, since a one-point wobble can flip which system looks better and bury a genuine quality regression under noise.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- G-Eval style judge prompts include scored exemplars per dimension, which is the canonical anchored-rubric pattern used across eval stacks in 2026.
- LangSmith and Braintrust let you attach few-shot scored examples to a judge so the rubric is anchored rather than abstract.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect that your anchor examples are biasing the judge's mean rather than just reducing variance?
Compare the anchored judge's score distribution against a human-labeled holdout. If variance dropped but the mean drifted away from human consensus, the anchors are skewed. Track both moments, not just spread.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming any anchor helps. Skewed anchors (all high-quality examples) calibrate the judge but anchor the mean upward, trading variance for a hidden bias you never measure.
60 second bullets to scan on the way to the call.
Why a bare 1 to 5 rubric produces unstable, idiosyncratic scoring
How anchor examples calibrate the judge by shared reference
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.