Which properties characterize a well-designed LLM eval rubric?

A **rubric** is the scoring guide you hand an LLM judge (or a human rater). Its whole job is to turn a fuzzy notion of quality into a number that means the same thing every time. Good rubric design is what separates a reliable eval from a random-number generator. Three of the options describe good design. First, separate the distinct quality aspects into their own dimensions. A RAG answer can be faithful to the source but off-topic, so you want one score for faithfulness and another for relevance, not one blurred average. Second, give an anchor example for each score level so '3 out of 5' means a concrete, shared thing rather than a private guess. Third, write concrete, observable criteria like 'cites at least one supporting fact' instead of vague adjectives like 'good' or 'thorough'. The other three options are anti-patterns. A single holistic score hides which dimension failed. Anchoring only the top and bottom leaves the whole middle of the scale undefined. And a rubric that adapts to response length quietly rewards verbosity, which is exactly the bias you are trying to remove.

A well-designed **rubric** does three things. It decomposes quality into independent dimensions. It anchors every scale point with a concrete example. It uses observable criteria, not adjectives. Dimension separation matters because quality is multi-axis. A faithful but off-topic answer should score high on faithfulness and low on relevance. A holistic score averages those into a meaningless middle number. Separate scores tell you which axis failed, so the eval is actionable. Anchoring matters because raters disagree most in the middle. If the prompt only describes a 5 and a 1, every judge invents its own 3. Pinning a worked example to each level cuts inter-rater variance sharply. Anchors are the single highest-leverage fix for judge noise. Observable criteria matter because adjectives are uninterpretable. The word thorough means different things to different judges. A predicate like cites at least one supporting fact is checkable and stable across runs. The three wrong options are classic traps. A holistic score conflates orthogonal axes. Anchoring only extremes leaves the middle undefined. And length-adaptive criteria systematically reward verbosity, baking in the exact bias good eval design fights.