Spot the design flaws in this LLM evaluation rubric

A judge **rubric** is the instruction sheet you give an LLM (or a human) telling it how to score a response. A good rubric makes two independent judges produce nearly the same score. This rubric fails on three counts. First, it has no anchor examples. It says score 1 for bad and 5 for excellent, but never shows what a 3 or a 4 looks like. Each judge invents its own definition of the middle, so the scores scatter. Second, it conflates three separate qualities into one number. Helpfulness, accuracy, and writing are independent. A response can be factually perfect but badly written. A single overall score cannot tell you which dimension failed. Third, the words 'bad' and 'excellent' are vague. They give the judge no concrete, actionable criteria, so different judges fill in different meanings. The fix: add a calibration example for every score level, score each dimension separately, and replace fuzzy adjectives with specific descriptions of what each level requires.

This rubric has three flaws that all hurt reliability. **No anchor examples.** It defines only the endpoints (1 = bad, 5 = excellent) and leaves 2, 3, and 4 undefined. Judges interpret the middle of any scale idiosyncratically. Without a concrete example for each level, inter-judge variance spikes. The fix is a calibration example per score point. **Conflated dimensions.** Helpfulness, accuracy, and writing quality are orthogonal. A response can be accurate but poorly written. Collapsing them into one holistic score hides which dimension failed and makes the number undiagnostic. The fix is to score each dimension on its own axis, then aggregate if needed. **Vague adjectives.** 'Bad' and 'excellent' are uninterpretable. Each judge supplies its own definition, so scores are not comparable across annotators. The fix is specific, actionable criteria stating what each level must demonstrate. Together these flaws mean two judges scoring the same response will disagree, and the resulting numbers are not trustworthy. A reliable rubric separates dimensions, anchors every level with an example, and replaces adjectives with observable criteria.