How does Chatbot Arena compute ELO rankings, and what bias affects its validity?

**Chatbot Arena** (now LMArena) is a public leaderboard where anonymous models battle head to head. You type a prompt, two unnamed models answer, and you click whichever reply you prefer. Millions of these votes get pooled into one ranking. The ranking is not a simple win percentage. The votes feed a statistical model called Bradley-Terry, which fits each model a single latent score. The bigger the gap between two scores, the more likely the higher model wins a given matchup. The result is published as an Elo-style number with a confidence interval, so you can see when two models are statistically tied. The headline number tells you which model the crowd prefers. It does NOT directly tell you which model is more accurate, more truthful, or better at your specific task. The main catch is who chooses the prompts. Users submit whatever interests them, which skews toward creative, tricky, or fun questions. So Arena rank reflects taste on a self-selected prompt mix, not uniform capability. Style and answer length also sway voters, which is why a chattier model can climb the board without being more correct.

**Chatbot Arena** turns anonymous pairwise human votes into a ranking. A user submits a prompt, two unnamed models reply, and the user picks the winner or a tie. The aggregation uses the Bradley-Terry model, the maximum-likelihood backbone behind Elo. Each model gets one latent score. Win probability is logistic: $$P(A \text{ beats } B) = \frac{e^{s_A}}{e^{s_A} + e^{s_B}} = \frac{1}{1 + e^{-(s_A - s_B)}}$$ Scores are fit by logistic regression over all recorded battles. The board reports each score with a bootstrap confidence interval, so overlapping intervals mean a statistical tie, not a real gap. The primary validity threat is query selection bias. Users self-select prompts toward creative or tricky cases. So the ranking measures crowd preference on a non-representative prompt mix, not uniform capability across tasks. Two further confounds matter. Voters favour longer, well-formatted answers, so style and verbosity inflate scores. And a model tuned for the Arena prompt distribution can outrank a broadly stronger one. This is why Arena rank can diverge from task accuracy on MMLU, coding, or your own eval set.