Elo is a pairwise rating system where wins and losses shift numeric scores, with larger shifts for upsets, converging to a reliable model ranking over many votes.
Imagine two basketball teams play a pickup game. If the team everyone expected to lose actually wins, the surprise is huge, so we move their ranking up a lot and the loser's ranking down a lot. If the expected winner wins, the rankings barely move because that was predictable. Now imagine thousands of these pickup games between different AI models, with real people voting on which response was better each time. After enough games, the rankings settle into a stable order that reflects how good each model actually is in practice. That is exactly how Elo works in Chatbot Arena.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Elo is one of those concepts that sounds like it belongs in chess tournaments, not AI engineering. But it has become the single most influential ranking mechanism for large language models, powering the Chatbot Arena leaderboard that researchers, journalists, and hiring managers all reference when they want to know which model is 'best' right now.
This deep dive explains the Elo system from first principles, shows how it applies to LLM evaluation, and identifies the places where the analogy between chess and chatbot ranking starts to creak.
Elo from first principles
The Elo system was invented by Arpad Elo in the 1960s for the US Chess Federation. The core idea is simple: every player holds a numeric rating, and each match updates both players' ratings based on the outcome.
The update formula is: new = old + K * (S - E), where S is the actual score (1 for win, 0.5 for draw, 0 for loss), E is the expected score (a logistic function of the rating gap), and K is a constant that controls how fast ratings move. The expected score is computed as:
The 400 in the denominator is a scaling constant: a 400-point gap means the stronger player is expected to win about 91% of the time. The logistic curve ensures that even a massive rating gap leaves some probability for an upset.
The key property is that upsets are informative. When a 1200-rated player beats a 1800-rated player, the update is large because the outcome was highly unlikely under the model. When a 1800 beats a 1200, the update is tiny because it was expected. This self-correcting behavior is what makes Elo converge to accurate rankings over time.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LMSYS Chatbot Arena is the most widely cited Elo leaderboard for LLMs in 2026, with hundreds of thousands of human votes across dozens of models.
- Chess platforms like Lichess and Chess.com use the same Elo system to rank millions of human players, which is where the algorithm originated.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does LMSYS decide which model pairs to show voters, and why does that matter for ranking accuracy?
Look into active sampling strategies. Prioritizing uncertain pairs (where the ranking is close or under-sampled) improves convergence speed and narrows confidence intervals faster than random pairing.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating Elo as an absolute quality score rather than a relative ranking that only makes sense within a specific comparison pool.
60 second bullets to scan on the way to the call.
State that Elo is a pairwise rating system with numeric scores
Explain the upset-weighting mechanism where surprising outcomes cause larger rating shifts
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.