Two candidates answer the same prompt. Response A: raw reward 6.0, 300 tokens. Response B: raw reward 4.5, 90 tokens. Under SimPO style length normalization (reward divided by token count), which response is preferred, and what are the two normalized scores?
B wins because reward density (4.5 / 90 = 0.05) beats A's diluted density (6.0 / 300 = 0.02), even though A's raw score is higher.
Imagine grading two essays. The first got 60 points but is 300 sentences long. The second got 45 points in only 90 sentences. If you ask which is better overall, you might say the first. But if you ask which earned more points per sentence, the second is way ahead, 0.5 versus 0.2. Length normalization is just that per-sentence view applied to AI responses. It stops the model from winning by piling on words. The math here is the same idea: divide each score by length and the shorter, denser answer wins.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This is a short arithmetic question wrapped around a long structural point. The numbers are easy: 0.02 versus 0.05, B wins. The interesting work is understanding why a single division operation is enough to dissolve one of the largest reward-hacking patterns in RLHF, and what new failure modes it introduces.
Length bias in RLHF arises because human annotators tend to prefer longer, more detailed-looking answers. The reward model absorbs that correlation. During PPO or DPO, the policy exploits it by padding. The remedy that emerged in 2024 and consolidated in 2026 practice is to score per token rather than per sequence, formalized in SimPO.
This deep dive walks through the arithmetic, the gradient geometry change that makes it work, the practical evaluation lever (length-controlled win rate), and the new failure modes that a length-normalized objective opens up. The goal is to make density-based scoring a tool you can defend and tune, not a magic word.
Doing the arithmetic and what it means
Reward density is just reward divided by length. For Response A, that is 6.0 / 300 = 0.02 per token. For Response B, 4.5 / 90 = 0.05 per token. B is 2.5x denser than A on this metric.
The raw versus density flip is the whole pedagogical point. A's raw advantage of 1.5 points looked decisive in absolute terms. Once you charge per token, A's extra 210 tokens become a cost rather than a free bonus, and the comparison inverts. The threshold for the flip is set by the length ratio: any time the longer response's reward to length ratio falls below the shorter response's, density wins.
A quick rule of thumb. If A is k times longer than B, A needs at least k times B's raw reward to beat B on density. Here A is 3.33x longer than B, so A would have needed at least 15.0 (3.33 * 4.5) to win. It only had 6.0. Density flips the call cleanly.
Do the arithmetic. Response A has total reward over tokens. Response B has over tokens. Raw reward picks B (3.6 > 2.4). Length-normalized reward picks A: vs . A is denser. The normalization changes the answer because it asks not which response scored higher overall, but which had more value per unit length. That is the operational definition of conciseness as a first-class objective rather than a side constraint.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- SimPO uses average per-token log-likelihood as the implicit reward, eliminating the length sensitivity DPO inherits from sequence-level scoring
- AlpacaEval's length-controlled (LC) win rate normalizes for response length before declaring a winner, and several Llama 3 variants jumped notably in LC ranking versus raw
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does DPO show length bias while SimPO does not, even with the same preference data?
Compare the two loss formulas at the token level. DPO compares sequence log-likelihoods so longer chosen responses move the loss more; SimPO averages per token so length cancels in the gradient.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Comparing raw rewards directly and concluding A wins. Length normalization specifically inverts that comparison when length differences are large.
60 second bullets to scan on the way to the call.
How to compute reward density given raw reward and token count
Why SimPO uses per-token average instead of sequence log-likelihood
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.