Which properties characterize a well designed LLM eval rubric?
Good rubrics decompose distinct dimensions, anchor every scale point with concrete examples, and use observable criteria. Holistic single scores, extreme-only anchors, and length-adaptive rules are anti-patterns.
Imagine judging a science fair. A bad judge just says 'that one felt good, 7 out of 10' and moves on. You learn nothing about why. A good judge uses a checklist: was the question clear? Was the method sound? Was the poster readable? Each gets its own score, so you know exactly where a project won or lost. Even better, the judge has sample posters pinned up showing what a 2, a 3, and a 4 actually look like, so every judge means the same thing by '3'. The worst judges only know what a perfect poster and a terrible poster look like, so everything in between is a guess. And a judge who gives more points just because a poster has more words is rewarding noise, not quality.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A rubric is the most underrated component of an LLM eval. The choice of judge model gets the attention, but the prompt's scoring guide determines whether the resulting numbers mean anything. A rubric is the operating manual for a noisy instrument: it tells the judge what to look at, how to grade it, and what each score level concretely means. Get it wrong and even a frontier judge like Claude Opus 4.7 produces numbers that wobble across runs and tell you nothing actionable.
The stakes are practical. An eval score is only useful if it is reproducible, interpretable, and trusted enough to gate a model swap or a prompt change. Each of those properties traces directly back to how the rubric is written. A vague rubric gives you a number that nobody believes; a sharp one gives you a number you can ship a release on.
This question contrasts three properties of a well-designed rubric against three plausible-sounding anti-patterns. The deep dive walks each good property, the failure mode it prevents, and the trap that mirrors it. The throughline is simple. Good rubric design reduces variance and removes bias; every wrong option quietly adds one or the other.
Dimension separation: quality is not scalar
The first good property is decomposing quality into independent dimensions, each scored on its own. A retrieval-augmented answer can be perfectly faithful to its source yet completely off-topic. It can be relevant yet incomplete. These axes are orthogonal, so collapsing them into one number destroys information.
Consider a faithful but off-topic answer. On a faithfulness axis it scores high; on a relevance axis it scores low. A holistic judge averages these into a misleading middle score that tells you nothing about which axis failed. Separate per-dimension scores make the eval actionable: you can see at a glance that faithfulness is fine and relevance is broken, and route the fix accordingly.
There is a second, subtler benefit. A single holistic score gives the judge latitude to anchor on overall fluency, which is precisely where length bias and self-preference bias slip in. Forcing the judge to grade narrow, named axes constrains that latitude and produces more stable numbers.
Dimension separation also unlocks weighting and aggregation. Once faithfulness and relevance are separate columns, you can weight them by what your product actually cares about, set per-dimension pass thresholds, and track each axis over time as a distinct regression signal. A blended score throws all of that away. This is exactly why frameworks like RAGAS expose faithfulness, answer relevance, and context precision as separate metrics rather than one number, and why teams build dashboards with one trend line per axis.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS scores faithfulness, answer relevance, and context precision as separate dimensions rather than one blended quality number.
- G-Eval prompts the judge for chain-of-thought reasoning against explicit per-criterion steps before emitting a score.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you quantify that anchor examples actually reduce judge variance?
Run the same judge twice on a fixed set with and without anchors, compute the standard deviation of repeated scores and inter-rater Cohen's kappa against humans, then compare the two conditions head to head.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Asking the judge for one holistic 1 to 10 score. It conflates orthogonal dimensions, hides which one failed, and produces noisy, unactionable numbers that drift across runs.
60 second bullets to scan on the way to the call.
Why decomposing dimensions beats a single holistic quality score
How anchor examples at each level cut inter-rater variance
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.