AlpacaEval 2 reports a 'win rate' for each model. Against what baseline and how is the winner decided?

**AlpacaEval 2** evaluates a model by comparing its responses against a fixed reference baseline on 805 instructions. For each instruction, an LLM judge reads both responses and decides which one is better. The win rate is simply the percentage of instructions where the evaluated model's response beats the baseline's response. The reference baseline is a strong model (GPT-4 Turbo in the standard configuration). This means the win rate is always relative to that specific baseline: a 30% win rate means the model beats the baseline on 30% of the 805 instructions. The length-controlled (LC) variant is a critical addition. LLM judges consistently prefer longer responses, even when shorter responses are equally good or better. This means a model can inflate its raw win rate just by being verbose. The LC correction adjusts for this bias, penalizing models whose wins are driven primarily by output length rather than actual quality improvement.

**AlpacaEval 2** evaluates models by comparing their outputs against a fixed reference baseline on 805 diverse instructions. An LLM judge decides the winner for each instruction, and the win rate is the percentage where the evaluated model's response is preferred. The design is intentionally simple. Rather than running a full round-robin tournament between all models, AlpacaEval compares every model against the same baseline. This makes adding a new model cheap (one evaluation run against the baseline) and ensures all scores are directly comparable since they share the same reference point. The length-controlled (LC) win rate is AlpacaEval 2's most important contribution. LLM judges exhibit a well-documented length bias: they prefer longer responses regardless of quality. A model that pads its answers with unnecessary detail can inflate its raw win rate without being genuinely better. The LC variant fits a regression model that separates the effect of output length from the effect of response quality, producing a debiased win rate that more accurately reflects actual improvement over the baseline. In practice, the gap between raw and LC win rates reveals how much of a model's advantage comes from verbosity versus genuine quality. Models with a large gap between the two are winning partly by being wordy, not by being better.