Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
A judge **rubric** is the instruction sheet you give an LLM (or a human) telling it how to score a response. A good rubric makes two independent judges produce nearly the same score. This rubric fails on three counts.
First, it has no anchor examples. It says score 1 for bad and 5 for excellent, but never shows what a 3 or a 4 looks like. Each judge invents its own definition of the middle, so the scores scatter.
Second, it conflates three separate qualities into one number. Helpfulness, accuracy, and writing are independent. A response can be factually perfect but badly written. A single overall score cannot tell you which dimension failed.
Third, the words 'bad' and 'excellent' are vague. They give the judge no concrete, actionable criteria, so different judges fill in different meanings.
The fix: add a calibration example for every score level, score each dimension separately, and replace fuzzy adjectives with specific descriptions of what each level requires.
This rubric has three flaws that all hurt reliability.
**No anchor examples.** It defines only the endpoints (1 = bad, 5 = excellent) and leaves 2, 3, and 4 undefined. Judges interpret the middle of any scale idiosyncratically. Without a concrete example for each level, inter-judge variance spikes. The fix is a calibration example per score point.
**Conflated dimensions.** Helpfulness, accuracy, and writing quality are orthogonal. A response can be accurate but poorly written. Collapsing them into one holistic score hides which dimension failed and makes the number undiagnostic. The fix is to score each dimension on its own axis, then aggregate if needed.
**Vague adjectives.** 'Bad' and 'excellent' are uninterpretable. Each judge supplies its own definition, so scores are not comparable across annotators. The fix is specific, actionable criteria stating what each level must demonstrate.
Together these flaws mean two judges scoring the same response will disagree, and the resulting numbers are not trustworthy. A reliable rubric separates dimensions, anchors every level with an example, and replaces adjectives with observable criteria.
Quick reference
TL;DR
This rubric has three design flaws: no anchor examples for middle scale levels, conflated dimensions collapsed into one score, and vague adjectives like 'bad' that judges interpret differently.
Imagine you ask ten teachers to grade the same essay from 1 to 5, but your only instruction is 'give a 1 if it is bad and a 5 if it is excellent.' One teacher thinks a 3 means average; another thinks 3 means barely passing. You never showed them what a real 3 looks like, so their scores scatter wildly. Worse, you mixed up three different questions into one number: is the essay helpful, is it factually correct, and is it well written? A response can nail the facts but read terribly, yet a single score cannot say which part failed. Finally, words like 'bad' mean different things to different people. Good rubrics show example answers for every score, grade each quality separately, and replace fuzzy words with concrete criteria.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks with a checkable gold answer (multiple-choice, structured extraction, math) Exact-match scoring is cheaper, unbiased, and needs no rubric. Skip the LLM judge.
Very early prototyping where you only need a rough directional signal A full multi-dimension anchored rubric is overkill before the product direction is settled.
Comparing two systems head to head rather than scoring on an absolute scale Pairwise comparison avoids absolute-scale calibration, so per-level anchors matter less than order randomisation.
Safety-critical or regulated evaluation where errors carry legal risk No rubric makes an LLM judge a safe sole signal. Human expert review is mandatory regardless of design.
Quick scan · 2–4 min
Everything important, quickly.
4 min: name the three flaws (anchorless levels, conflated dimensions, vague adjectives), explain why each breaks reproducibility, give the fix for each, then tie to validating the rubric against human labels.
Real-world examples
Real products, models, and research that use this idea.
1RAGAS scores faithfulness, answer relevance, and context precision as separate dimensions rather than one holistic number.
2G-Eval prompts the judge with explicit per-step criteria and chain-of-thought rather than a bare 1 to 5 instruction.
3Prometheus 2 is trained on rubric plus reference answer prompts precisely because anchored rubrics beat free-form scoring.
4LangSmith evaluator templates expose per-criterion scoring and let you attach few-shot anchor examples per level.
5Chatbot Arena uses pairwise human votes to sidestep absolute-scale calibration entirely.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide between an absolute 1 to 5 rubric and pairwise comparison?
Sign in or go Pro to see the approach for this follow-up.
QHow do you measure whether your fixed rubric actually improved judge reliability?
Sign in or go Pro to see the approach for this follow-up.
QWhy might separating dimensions sometimes hurt rather than help?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating a holistic 1 to 5 score with no anchor examples as a reliable signal. Vague adjectives and conflated dimensions produce scores that vary across judges and cannot diagnose which quality failed.