Predict which answer a vanilla GPT-4 judge prefers: short+correct vs long+off topic

Predict output·Hard·4.0 · 0·~2 min·Asked atFiddler Ai Fireworks Ai Vernacular Ai·Relevant atAnthropic

Attempt it

A vanilla GPT-4 judge (no anti length instructions) is evaluating two responses to the question: 'What year did the French Revolution begin?'

Response A (short, correct): 'The French Revolution began in 1789.'

Response B (long, slightly off topic): 'The French Revolution, one of the most transformative events in European history, is generally considered to have begun in 1789 with a series of tumultuous events including the storming of the Bastille on July 14th. However, historians sometimes debate whether the seeds were planted earlier in the 1780s with the financial crisis, the Estates General convening, and mounting social unrest. The revolution itself continued in various phases until approximately 1799 with Napoleon's rise to power. Its causes, consequences, and legacy continue to be studied by scholars worldwide.'

Which response will the judge most likely prefer, and why?

The judge most likely prefers **Response B**, the long one. This is the classic trap: the answer the judge picks is not the answer a careful human would pick. Both responses are factually correct. Response A says the French Revolution began in 1789, which is exactly the answer to the question asked. Response B also says 1789, but wraps it in paragraphs about the Bastille, the 1780s financial crisis, and Napoleon, most of which the question never asked for. The reason the judge favors B is **length bias**. LLM judges are usually trained on top of models tuned with human feedback, and human annotators tended to rate longer, more detailed answers higher. The model learned a shortcut: more words, more structure, and academic-sounding context signal a better answer. That shortcut is wrong here. Response B is partly off topic and adds nothing the question needed. A vanilla judge with no instruction to ignore length cannot tell the difference between genuine thoroughness and padding. The fix is to tell the judge what to value. Add explicit rubric instructions like reward correctness and conciseness, penalize irrelevant content, or compare answers of matched length so the judge cannot lean on the word count.

The judge prefers **Response B**, the longer answer. This is length bias, also called verbosity bias. Both answers contain the correct year, 1789. Response A is short and precise. Response B is long and partly off topic. A careful human grader would prefer A for this narrow factual question. A vanilla judge does the opposite. The mechanism is learned, not random. Judge models inherit preferences from RLHF training. Human raters systematically scored longer, more detailed answers higher. So the judge learned to associate length, structure, and academic phrasing with quality. Response B triggers every one of those surface signals. It has multi-sentence elaboration, historical context, and a confident scholarly tone. None of that improves the answer to what year, but the judge cannot separate real substance from fluent padding. The interview point is the asymmetry. Length bias makes the judge reward verbosity over precision. Without an anti-length instruction, the judge optimizes for what looks thorough, not what is correct and on point. The mitigations are concrete. Write a rubric that explicitly rewards conciseness and penalizes irrelevance. Normalize for length, or compare matched-length candidates. Validate the judge against human labels and watch the case where it disagrees on short correct answers.

TL;DR

The judge prefers Response B, the long one. Length bias makes RLHF-trained judges read verbosity as thoroughness, so they reward the padded answer over the short precise one.

Memory aid

Easy to grasp

Imagine a teacher grading two students who both got the answer right. The first writes 'The answer is 1789.' The second writes a full page about the French Revolution, the Bastille, Napoleon, and what historians still argue about. Even though both nailed the actual question, many teachers instinctively give the long answer a better grade because it LOOKS like more effort and more knowledge. An LLM judge does the same thing. It was trained on human ratings where longer, more detailed replies usually scored higher, so it learned a shortcut: more words equals better. Here that shortcut backfires. Response A is exactly correct and on point. Response B drifts off topic. But the judge, without being told to ignore length, still picks B.

Key concepts

LLM-as-judge Length bias Verbosity bias Rubric design

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

This question looks like a trivia check but is really a test of whether you understand how an LLM judge actually behaves versus how you wish it behaved. Both responses contain the correct year, 1789, so neither is factually wrong. The trap is that the question asks what the judge will DO, not what a thoughtful human grader SHOULD do. Candidates who answer with the better response, A, have correctly judged the answers but have failed to predict the judge, which is the actual task.

The predicted output is Response B, the long one. A vanilla judge with no anti-length instruction prefers the verbose, partly off-topic answer over the short, precise one. This is not a quirk of one prompt or one model; it is a structural property of how judge models are built. The rest of this walkthrough explains the mechanism behind that prediction, why it is reliable rather than a coin flip, how it corrupts real evaluation pipelines, and the concrete mitigations that neutralize it. Treat the judge throughout as what it really is: a measurement instrument with a known, directional systematic error.

Why the judge picks B: length bias is learned

LLM judges are not neutral oracles. They are language models, usually ones tuned with reinforcement learning from human feedback, repurposed to score other models. During that human feedback stage, annotators consistently rated longer, more detailed, more hedged answers higher than terse ones. Length is an easy proxy for effort, and effort correlates with quality often enough that raters lean on it. That preference becomes a learned prior baked into the reward model and then into the policy weights.

When you then ask the model to judge, it carries that prior with it. It has internalized a correlation: longer and more structured tends to mean better. This is what the literature calls length bias or verbosity bias, and it is one of the most reproducible effects in the LLM-as-judge space. Critically, the model has no separate module that says this is judging mode, turn off your generation preferences. The same internal preferences that make it generate verbose answers make it reward verbose answers.

Response B is engineered, accidentally, to trigger every signal the judge associates with quality. It has multiple sentences, an academic vocabulary, historical scaffolding, hedging phrases like historians sometimes debate, and a confident scholarly tone. Each of those is a feature the reward model learned to upvote. The judge reads them as evidence of a thorough, high-quality answer, even though the question only asked for a year. Response A, by contrast, presents none of those upvoted features. It is correct and complete for the task, but it offers the judge nothing to reward beyond the bare fact.

Correctness is not the same as response quality

Why this is predictable, not random

How length bias corrupts real evaluation pipelines

Mitigations: rubric, length control, decomposition

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

AlpacaEval 2.0 added length-controlled win rates in 2024 specifically to neutralize the verbosity bias this question demonstrates.
MT-Bench and Chatbot Arena documented length bias as a core confound in LLM-judge experiments.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you experimentally confirm a judge has length bias rather than just preferring better answers?

Hold quality fixed and vary only length: take a correct answer and pad it with on-topic but redundant filler. If the padded version wins consistently, the signal is length, not substance. AlpacaEval's length-controlled metric formalizes this.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming the judge rewards the more accurate answer. A vanilla judge optimizes perceived thoroughness, so it picks the longer reply even when the short one is more precise.

60 second bullets to scan on the way to the call.

Why a vanilla judge prefers the longer answer here
How RLHF training installs the length to quality association

Primary sources. Browse if you want the original framing.