A rubric is a structured scoring guide that defines what each score level means with concrete criteria and anchor examples. Without a rubric, judges (human or LLM) interpret quality subjectively, producing inconsistent and non-reproducible scores. A good rubric separates dimensions (relevance, accuracy, completeness) and anchors each level with a real example so that different judges converge on the same score for the same output.
A **rubric** is a structured document that tells judges exactly what each score level means for each quality dimension you care about. It is the foundation of any LLM-as-judge or human evaluation setup.
A typical rubric has three components. First, dimensions: the independent aspects of quality you want to evaluate, such as relevance, factual accuracy, and completeness. Second, score levels: a numeric scale (usually 1 to 5) with a written definition for each level on each dimension. Third, anchor examples: real model outputs that illustrate each score level, so judges can compare their current output to a concrete reference.
Without a rubric, 'rate this output from 1 to 5' means something different to every judge. One judge interprets 3 as acceptable, another as mediocre. One focuses on grammar, another on factual accuracy. The resulting scores are noisy and non-reproducible.
With a rubric, every judge applies the same criteria. A score of 3 on accuracy means 'the answer contains the core fact but includes one minor inaccuracy.' A score of 5 means 'the answer is fully correct with no factual errors.' Judges can disagree about edge cases, but the rubric constrains the range of disagreement enough to produce actionable data.
In LLM-as-judge evaluation, a **rubric** is a structured scoring guide that defines quality dimensions, score levels, and anchor examples so that judges (human or LLM) produce consistent, reproducible scores.
A well-designed rubric has three layers. The first layer names the dimensions being evaluated: relevance, factual accuracy, completeness, coherence, or whatever matters for the task. Each dimension is scored independently so that a single weak dimension does not get masked by strong performance on others.
The second layer defines score levels for each dimension. A 5-point scale is standard. Each level gets a concrete definition: not 'good' or 'bad' but observable criteria. For accuracy, level 5 might be 'all factual claims are correct and verifiable,' level 3 might be 'core claim is correct but one secondary detail is wrong,' and level 1 might be 'the central factual claim is incorrect.'
The third layer provides anchor examples: real model outputs that have been scored by experts and serve as calibration points. When a judge is uncertain whether an output is a 3 or a 4, they compare it to the anchor examples and pick the closer match. This reduces subjective drift dramatically.
For LLM judges specifically, the rubric is embedded in the system prompt. The judge model reads the rubric, reads the output, and returns a score with reasoning. Without a rubric in the prompt, the LLM judge relies on its own implicit quality model, which is inconsistent across calls and biased by its training data. The rubric constrains the judge to evaluate what you care about, not what it defaults to.
Quick reference
TL;DR
A rubric is a structured scoring guide with defined criteria and anchor examples for each score level, ensuring consistent and reproducible evaluation by human or LLM judges.
Imagine three teachers grading the same essay. Without a rubric, one teacher cares about grammar, another about creativity, and the third about whether the essay answers the question. They give the same essay three different grades. A rubric fixes this. It says: grade on three dimensions (relevance, accuracy, grammar). For each dimension, score 1 means this specific thing, score 3 means this specific thing, and score 5 means this specific thing. Here is an example essay for each score level so you can compare. Now all three teachers are grading the same way, and you can trust that the scores mean something consistent. In LLM evaluation, the judges might be humans or other LLMs, but the rubric serves the same purpose. It turns subjective opinions into reproducible measurements.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Binary pass/fail evaluation where the output either meets a clear specification or does not A rubric with multiple score levels adds unnecessary complexity when the decision is binary. A simple pass/fail criterion suffices.
Automated metric evaluation where scoring is algorithmic (BLEU, ROUGE, pass@k) These metrics do not involve subjective judgment, so a rubric for human or LLM judges adds nothing. The algorithm IS the scoring guide.
Early rapid prototyping where you are generating a handful of outputs and reviewing them yourself A formal rubric is overhead when one person is informally assessing feasibility. Build the rubric once you move to systematic evaluation.
Quick scan · 2–4 min
Everything important, quickly.
5 min: define rubric with its three components (dimensions, levels, anchors), explain why rubrics are required for LLM-as-judge consistency, name the three main judge biases rubrics mitigate, and close with inter-rater agreement as the rubric quality metric.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's internal RLHF process uses rubrics to train human raters who generate preference data, ensuring consistency across hundreds of annotators rating Claude's outputs.
2Promptfoo supports custom rubric definitions in its LLM-judge evaluators, letting teams embed scoring criteria directly in their eval config files.
3MT-Bench uses a structured rubric that instructs the LLM judge to evaluate multi-turn responses on helpfulness, relevance, accuracy, depth, and creativity.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you know if your rubric is good enough? What signals tell you the rubric needs revision?
Sign in or go Pro to see the approach for this follow-up.
QHow do you embed a rubric in an LLM-judge prompt so the judge actually follows it?
Sign in or go Pro to see the approach for this follow-up.
QDoes using a rubric fully eliminate LLM-judge biases?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Writing a rubric with vague criteria like 'the answer is good' instead of concrete, observable requirements. Vague rubrics produce the same inconsistency as having no rubric at all.