What is the primary validity threat to Chatbot Arena ELO rankings as a measure of general model quality?

**Chatbot Arena** (now LMArena) ranks models by crowdsourced human preference. A user types a prompt, two anonymous models answer, the user votes for the better one, and identities are revealed afterward. Millions of these votes turn into a single rating per model. The correct answer is query selection bias. The people who use the Arena choose their own prompts. That means the leaderboard reflects what those particular users like to ask about, not a balanced sample of every task a model might face in the real world. A model tuned for open-ended chat, coding, and creative writing can top the board, while a model better at niche enterprise or domain workflows looks worse, simply because few people submit those prompts. The other options are distractors. The Bradley-Terry math behind the rating is well-understood and not the core flaw. Vote manipulation is constrained by the platform and is not the headline threat. And Elo generalizes fine to pairwise preference, that is exactly what it models. The takeaway: Arena measures real human preference on a skewed sample of prompts. Treat it as one strong signal, not a complete capability score.

**Chatbot Arena** crowdsources pairwise human preference. Users submit prompts, vote between two anonymous model outputs, and a rating is fit to the win-loss record. The primary validity threat is query selection bias. Users self-select which prompts to submit. So the prompt distribution reflects the Arena's user population, not a uniform sample of capability. A model that wins on those self-selected prompts ranks high. It may still lag on enterprise workflows or domain tasks that are underrepresented. The rating math is not the weak point. Win probability follows a Bradley-Terry model, which Elo approximates online: $$P(A \succ B) = \frac{1}{1 + 10^{(R_B - R_A)/400}}$$ This estimator is well-specified and converges with enough comparisons. The distractor about independence of irrelevant alternatives names a real Bradley-Terry assumption, but it is not the main practical threat. The genuine strengths are worth stating. Arena captures real human preference, not a proxy metric. And it is hard to game, since prompts are live and identities hidden during voting. The genuine limit is twofold. Preference is not correctness, voters reward fluent and confident answers. And the sample skews toward whoever shows up. Treat Arena as one strong axis among several, never as ground truth.