What is the primary advantage of chain-of-thought (CoT) judge prompts over direct-score prompts?

A **chain-of-thought judge** is an LLM evaluator told to write out its reasoning before committing to a score. A direct-score judge just emits the number. The difference sounds cosmetic but it changes the quality of the scores. The primary advantage is reliability on hard cases. When a judge outputs a number immediately, it tends to anchor on the first salient thing it notices, a confident tone or a strong opening, and the rest of the answer barely moves the score. Forcing the judge to reason through each rubric dimension first spreads its attention across all the criteria, so the final score reflects the whole answer rather than a snap impression. There is a second bonus: the reasoning trace is auditable. You can read why the judge scored a 3 and decide whether you trust it, which you cannot do with a bare number. The three wrong options are tempting distractors. CoT is more expensive, not cheaper, because reasoning tokens add up. It does not eliminate position bias; the bias can still leak into the reasoning. And it does not give the judge external knowledge, so factual errors persist if the judge never knew the fact.

**Chain-of-thought judging** asks the model to reason through the rubric before emitting a score. The primary win is higher reliability on ambiguous, multi-criterion cases. A direct-score judge anchors. It latches onto the first salient feature, fluency or confidence, and the number reflects that snap impression. Forcing explicit reasoning makes the judge touch every rubric dimension before it commits. That raises agreement with human labels and consistency across runs. The reasoning trace is also auditable, so you can debug a suspicious score. The gain is largest where direct scoring is weakest: long answers, subjective rubrics, and near-ties. The cost side matters. CoT spends extra output tokens on reasoning, so it is more expensive, not cheaper. The other distractors are also wrong. CoT does not eliminate position bias; the bias can still surface inside the reasoning itself. And it grants no external knowledge, so a judge that never knew a fact still scores a confident falsehood highly. One real risk is rationalization. The judge can write plausible reasoning that justifies a predetermined score rather than deriving the score from the reasoning. Anchor examples and forced per-dimension scoring help curb that.