Which properties characterize a well-designed LLM eval rubric?

Attempt it

Quick reference

TL;DR

Good rubrics decompose distinct dimensions, anchor every scale point with concrete examples, and use observable criteria. Holistic single scores, extreme-only anchors, and length-adaptive rules are anti-patterns.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine judging a science fair. A bad judge just says 'that one felt good, 7 out of 10' and moves on. You learn nothing about why. A good judge uses a checklist: was the question clear? Was the method sound? Was the poster readable? Each gets its own score, so you know exactly where a project won or lost. Even better, the judge has sample posters pinned up showing what a 2, a 3, and a 4 actually look like, so every judge means the same thing by '3'. The worst judges only know what a perfect poster and a terrible poster look like, so everything in between is a guess. And a judge who gives more points just because a poster has more words is rewarding noise, not quality.

Key concepts

Eval prompt design Rubric anchoring Dimension separation LLM-as-judge

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: three good design properties (dimension separation, level anchoring, observable criteria) plus three traps (holistic blur, extreme-only anchors, length adaptation), with the calibration loop that ties it together.

Real-world examples

Real products, models, and research that use this idea.

RAGAS scores faithfulness, answer relevance, and context precision as separate dimensions rather than one blended quality number.
G-Eval prompts the judge for chain-of-thought reasoning against explicit per-criterion steps before emitting a score.
Prometheus 2 is trained on rubric plus anchor prompts and ships score-level descriptors for each point on the scale.
LangSmith and Braintrust evaluator templates expose per-dimension scoring with example-anchored levels as the default pattern.
OpenAI Evals model-graded templates encourage concrete pass criteria over holistic vibes for reproducible scoring.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you quantify that anchor examples actually reduce judge variance?

Sign in or go Pro to see the approach for this follow-up.

QWhen does adding more rubric dimensions start to hurt rather than help?

Sign in or go Pro to see the approach for this follow-up.

QWhy is requiring chain-of-thought before the score a robustness fix, not just verbosity?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Asking the judge for one holistic 1 to 10 score. It conflates orthogonal dimensions, hides which one failed, and produces noisy, unactionable numbers that drift across runs.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.