Before running LLM-as-judge, someone insists on writing a rubric first. What is a rubric in this context?

A **rubric** is a structured document that tells judges exactly what each score level means for each quality dimension you care about. It is the foundation of any LLM-as-judge or human evaluation setup. A typical rubric has three components. First, dimensions: the independent aspects of quality you want to evaluate, such as relevance, factual accuracy, and completeness. Second, score levels: a numeric scale (usually 1 to 5) with a written definition for each level on each dimension. Third, anchor examples: real model outputs that illustrate each score level, so judges can compare their current output to a concrete reference. Without a rubric, 'rate this output from 1 to 5' means something different to every judge. One judge interprets 3 as acceptable, another as mediocre. One focuses on grammar, another on factual accuracy. The resulting scores are noisy and non-reproducible. With a rubric, every judge applies the same criteria. A score of 3 on accuracy means 'the answer contains the core fact but includes one minor inaccuracy.' A score of 5 means 'the answer is fully correct with no factual errors.' Judges can disagree about edge cases, but the rubric constrains the range of disagreement enough to produce actionable data.

In LLM-as-judge evaluation, a **rubric** is a structured scoring guide that defines quality dimensions, score levels, and anchor examples so that judges (human or LLM) produce consistent, reproducible scores. A well-designed rubric has three layers. The first layer names the dimensions being evaluated: relevance, factual accuracy, completeness, coherence, or whatever matters for the task. Each dimension is scored independently so that a single weak dimension does not get masked by strong performance on others. The second layer defines score levels for each dimension. A 5-point scale is standard. Each level gets a concrete definition: not 'good' or 'bad' but observable criteria. For accuracy, level 5 might be 'all factual claims are correct and verifiable,' level 3 might be 'core claim is correct but one secondary detail is wrong,' and level 1 might be 'the central factual claim is incorrect.' The third layer provides anchor examples: real model outputs that have been scored by experts and serve as calibration points. When a judge is uncertain whether an output is a 3 or a 4, they compare it to the anchor examples and pick the closer match. This reduces subjective drift dramatically. For LLM judges specifically, the rubric is embedded in the system prompt. The judge model reads the rubric, reads the output, and returns a score with reasoning. Without a rubric in the prompt, the LLM judge relies on its own implicit quality model, which is inconsistent across calls and biased by its training data. The rubric constrains the judge to evaluate what you care about, not what it defaults to.