Chatbot Arena
Also known as: LMSYS Arena, LMArena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
A crowdsourced LLM evaluation platform where humans pick between two anonymized model responses to the same prompt. Pairwise outcomes are aggregated into an Elo rating that ranks models by perceived quality.
In practice
Most-cited human-preference leaderboard. Expect questions on its bias (style preference, prompt distribution) and why model trainers chase it.
Related topics
Questions that mention this term
- Why does Chatbot Arena (LMSYS) give a different picture of model quality than static benchmarks like MMLU?
- Chatbot Arena ranks models by Elo rating. Explain the Elo system to someone who has never played competitive chess.
- What is the primary validity threat to Chatbot Arena ELO rankings as a measure of general model quality?
- How does Chatbot Arena compute ELO rankings, and what bias affects its validity?
- MT-Bench is used alongside Chatbot Arena. Describe what MT-Bench tests that single turn benchmarks miss.
Related terms
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.
Jailbreak
Adversarial prompt that bypasses an LLM's safety training to elicit normally-refused content.