A **scoring rubric** is a document that tells the judge exactly what each score level means. Without it, a '4 out of 5' is meaningless because different judges interpret the scale differently.
A well-designed rubric has three parts. First, it names the quality dimensions being evaluated: accuracy, helpfulness, completeness, tone. Second, it defines what each score level means for each dimension. For example: '5 = fully correct, complete, and well-structured. 3 = mostly correct but missing important details. 1 = factually wrong or irrelevant.' Third, it includes anchor examples: concrete model outputs that illustrate what a 1, 3, and 5 actually look like.
The rubric matters because it is what turns a subjective 'how good is this?' into a structured task. Without it, one judge might focus on accuracy while another focuses on tone, and both assign the same numeric score meaning different things. With a rubric, both judges evaluate the same dimensions using the same criteria.
This applies equally to human judges and LLM judges. When you use an LLM as a judge, the rubric is embedded in the eval prompt template. The judge reads the criteria and anchor examples before scoring, which grounds its judgment.
A **scoring rubric** defines what each score level (e.g., 1 through 5) means with explicit criteria and ideally anchor examples. It turns a vague 'rate quality' task into a structured judgment that different judges can apply consistently.
Without a rubric, two judges can both give a '4' but mean completely different things. One might mean 'accurate but incomplete.' The other might mean 'complete but has a tone issue.' The number is the same, but the signal is different. A rubric eliminates this ambiguity by specifying what each number means on each dimension.
Anchor examples are the highest-leverage component. Instead of just describing a score level in abstract terms, you show the judge a concrete output that deserves that score. Anchors reduce interpretation variance because the judge has a reference point to compare against.
In LLM-as-judge pipelines, the rubric is embedded in the eval prompt template. The judge reads the criteria and anchors before evaluating. Research shows that rubrics with anchor examples produce higher inter-rater agreement than rubrics with criteria descriptions alone. The G-Eval approach goes further by adding chain-of-thought reasoning: the judge evaluates each criterion explicitly before assigning a final score.
The practical discipline: never deploy an LLM judge without a rubric. A judge without a rubric is making up its own criteria, and those criteria shift with prompt context, model version, and input distribution.
Quick reference
TL;DR
A scoring rubric defines explicit criteria for each score level, ideally with anchor examples, so different judges produce consistent evaluations instead of arbitrary numbers.
Imagine you and a friend are both judging a sandcastle competition. Without any rules, you might give a castle a 4 because it is tall, while your friend gives it a 4 because it is detailed. Your 4s mean different things. A rubric fixes this. It says: '5 means the castle has multiple towers, a moat, and fine detail work. 3 means it has basic shape but no detail. 1 means it collapsed.' Now when either of you gives a 4, it means the same thing. In LLM evaluation, a scoring rubric does the same job: it tells the judge (human or AI) exactly what each score level looks like, so everyone grades on the same scale.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Binary pass/fail evaluation with a clear ground truth When correctness is unambiguous (code passes tests, classification matches label), a rubric adds unnecessary complexity. The evaluation is already structured by the ground truth.
Pairwise comparison where the judge only picks a winner Pairwise evaluation asks 'which is better?' not 'what score does each deserve?' A preference rubric can help define 'better,' but a full scoring rubric is overkill.
Automated metrics like BLEU or ROUGE that compute overlap mathematically These metrics have fixed formulas. There is no judge to instruct and no score level to define. The rubric concept applies to human and LLM judges, not deterministic metrics.
Quick scan · 2–4 min
Everything important, quickly.
5 min: define the rubric, explain why it matters for consistency, describe anchor examples as the key component, connect to the eval prompt template, mention chain-of-thought, and close with the versioning discipline.
Real-world examples
Real products, models, and research that use this idea.
1MT-Bench defines a 1 to 10 scoring rubric that instructs the LLM judge on what constitutes a low, medium, and high-quality multi-turn response.
2Promptfoo lets teams define custom rubrics per evaluation dimension, with criteria and optional anchor examples embedded in the eval prompt template.
3G-Eval demonstrated that adding chain-of-thought reasoning to the rubric evaluation process improved judge-human correlation on summarization quality.
4Braintrust AI supports rubric versioning so teams can track how scoring criteria evolve and re-calibrate judges when criteria change.
5Human evaluation guidelines at Chatbot Arena include a rubric that instructs voters on what 'better' means in a pairwise comparison.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow many anchor examples per score level produce the best judge consistency?
Sign in or go Pro to see the approach for this follow-up.
QHow do you handle a rubric dimension where human experts disagree on what the score levels should mean?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens to historical evaluation data when you change the rubric?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Asking a judge to 'rate quality from 1 to 5' without defining what each score means. Without a rubric, scores are arbitrary and inconsistent across judges.