Bradley-Terry expects a real preference. Force a winner on a tie and you inject noise or, worse, bake in length and position biases.
Imagine asking two people to taste two glasses of the same wine and pick the better one. Pressing them to pick something teaches you nothing about wine; it teaches you about which side they tend to favour, or maybe which glass they tasted first. Real ties are real information: they tell you the two responses are equally good (or equally bad) and that the comparison is not useful for ranking. The right move is to drop them or treat them honestly with a tie-aware loss. Never force a winner; you will just train the model on whatever bias decided the coin flip.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tie handling in preference data is a small-looking design choice with large downstream consequences. The Bradley-Terry loss that powers standard RM training assumes one response is genuinely preferred; feeding it ties as if they were real preferences corrupts the signal. The corruption can be small (random forced winners just add noise) or systemic (length and position tiebreaks bake structural biases directly into the RM).
Option A is the only answer that respects what a tie means and avoids the systemic-bias trap. The other options each pick a different way to silently force a winner, each of which has a specific downstream failure mode.
Mental model: a tie is information, not missing data. It tells you the two responses are equally good (or equally bad) and the comparison carries no ranking signal. Drop it or model it explicitly; never invent a winner.
Why Bradley-Terry breaks on forced ties
The Bradley-Terry loss. The standard RM training objective is a contrastive log-likelihood on preference pairs:
The loss assumes a winner y_w and a loser y_l. The gradient pushes the winner's score above the loser's score by an amount controlled by the sigmoid saturation.
What happens on a true tie. If the two responses are genuinely equally preferred, there is no correct gradient direction. Any forced winner produces a gradient that either:
- Cancels in expectation over many such pairs (random forcing) but adds variance everywhere.
- Aligns with the forcing rule (systematic forcing) and trains the RM to encode the rule itself.
The information cost of dropping ties is small. A tie carries no ranking information by definition. The information loss from filtering it out is essentially zero on the ranking dimension. There is some signal in 'these two responses are similar in quality' that a tie-aware loss could extract, but it is small and not always worth the implementation complexity.
The information cost of forcing a winner is large. Random forcing adds variance to every tie pair, which propagates into noisier RM scores in regions near the decision boundary. Systematic forcing adds a coherent bias signal that the RM learns and the policy later exploits.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's InstructGPT preference-data pipeline drops tie pairs from Bradley-Terry training; the practice is documented in the published methodology.
- Anthropic's RLHF stack and constitutional RLAIF pipelines treat ties as an informative meta-signal; high tie rates trigger rubric or category review.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a tie-aware loss that extends Bradley-Terry while remaining compatible with existing RM training infrastructure?
The simplest is a weighted soft target: for tie pairs, target sigmoid output 0.5 with a reduced loss weight (say 0.5x). The implementation change is one line; the model learns 'these are similar' from ties without being told one is preferred. Rao-Kupper is more principled but requires per-pair tie-probability parameters.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forcing a winner on a true tie injects label noise at best and bakes in length or position bias at worst; the seemingly simple choice corrupts the entire RM.
60 second bullets to scan on the way to the call.
Why Bradley-Terry assumes a real preference and breaks on forced ties
The two systematic-tiebreak failure modes: length inflation and position-bias amplification
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.