Before running LLM-as-judge, someone insists on writing a rubric first. What is a rubric in this context?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

A rubric is a structured scoring guide that defines what each score level means with concrete criteria and anchor examples. Without a rubric, judges (human or LLM) interpret quality subjectively, producing inconsistent and non-reproducible scores. A good rubric separates dimensions (relevance, accuracy, completeness) and anchors each level with a real example so that different judges converge on the same score for the same output.

Your answer

A **rubric** is a structured document that tells judges exactly what each score level means for each quality dimension you care about. It is the foundation of any LLM-as-judge or human evaluation setup. A typical rubric has three components. First, dimensions: the independent aspects of quality you want to evaluate, such as relevance, factual accuracy, and completeness. Second, score levels: a numeric scale (usually 1 to 5) with a written definition for each level on each dimension. Third, anchor examples: real model outputs that illustrate each score level, so judges can compare their current output to a concrete reference. Without a rubric, 'rate this output from 1 to 5' means something different to every judge. One judge interprets 3 as acceptable, another as mediocre. One focuses on grammar, another on factual accuracy. The resulting scores are noisy and non-reproducible. With a rubric, every judge applies the same criteria. A score of 3 on accuracy means 'the answer contains the core fact but includes one minor inaccuracy.' A score of 5 means 'the answer is fully correct with no factual errors.' Judges can disagree about edge cases, but the rubric constrains the range of disagreement enough to produce actionable data.

In LLM-as-judge evaluation, a **rubric** is a structured scoring guide that defines quality dimensions, score levels, and anchor examples so that judges (human or LLM) produce consistent, reproducible scores. A well-designed rubric has three layers. The first layer names the dimensions being evaluated: relevance, factual accuracy, completeness, coherence, or whatever matters for the task. Each dimension is scored independently so that a single weak dimension does not get masked by strong performance on others. The second layer defines score levels for each dimension. A 5-point scale is standard. Each level gets a concrete definition: not 'good' or 'bad' but observable criteria. For accuracy, level 5 might be 'all factual claims are correct and verifiable,' level 3 might be 'core claim is correct but one secondary detail is wrong,' and level 1 might be 'the central factual claim is incorrect.' The third layer provides anchor examples: real model outputs that have been scored by experts and serve as calibration points. When a judge is uncertain whether an output is a 3 or a 4, they compare it to the anchor examples and pick the closer match. This reduces subjective drift dramatically. For LLM judges specifically, the rubric is embedded in the system prompt. The judge model reads the rubric, reads the output, and returns a score with reasoning. Without a rubric in the prompt, the LLM judge relies on its own implicit quality model, which is inconsistent across calls and biased by its training data. The rubric constrains the judge to evaluate what you care about, not what it defaults to.

Quick reference

TL;DR

A rubric is a structured scoring guide with defined criteria and anchor examples for each score level, ensuring consistent and reproducible evaluation by human or LLM judges.

Memory aid

Easy to grasp

Imagine three teachers grading the same essay. Without a rubric, one teacher cares about grammar, another about creativity, and the third about whether the essay answers the question. They give the same essay three different grades. A rubric fixes this. It says: grade on three dimensions (relevance, accuracy, grammar). For each dimension, score 1 means this specific thing, score 3 means this specific thing, and score 5 means this specific thing. Here is an example essay for each score level so you can compare. Now all three teachers are grading the same way, and you can trust that the scores mean something consistent. In LLM evaluation, the judges might be humans or other LLMs, but the rubric serves the same purpose. It turns subjective opinions into reproducible measurements.

Key concepts

Rubric Scoring Guide Anchor Examples Score Levels Consistency

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define rubric with its three components (dimensions, levels, anchors), explain why rubrics are required for LLM-as-judge consistency, name the three main judge biases rubrics mitigate, and close with inter-rater agreement as the rubric quality metric.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's internal RLHF process uses rubrics to train human raters who generate preference data, ensuring consistency across hundreds of annotators rating Claude's outputs.
Promptfoo supports custom rubric definitions in its LLM-judge evaluators, letting teams embed scoring criteria directly in their eval config files.
MT-Bench uses a structured rubric that instructs the LLM judge to evaluate multi-turn responses on helpfulness, relevance, accuracy, depth, and creativity.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you know if your rubric is good enough? What signals tell you the rubric needs revision?

QHow do you embed a rubric in an LLM-judge prompt so the judge actually follows it?

QDoes using a rubric fully eliminate LLM-judge biases?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Writing a rubric with vague criteria like 'the answer is good' instead of concrete, observable requirements. Vague rubrics produce the same inconsistency as having no rubric at all.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.

Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena