Every model release cites benchmark numbers. Define what a benchmark actually is and name two things it cannot tell you.

A **benchmark** is a standardized evaluation: a fixed set of tasks with a predetermined scoring method that lets anyone compare model performance on equal terms. The key word is standardized. Everyone uses the same inputs, the same scoring function, and the same split between test and development data. This makes scores directly comparable across labs, model versions, and time. When Anthropic reports that Claude scores 87% on MMLU-Pro and OpenAI reports GPT-5.5 scores 85%, you know both numbers were computed on the same test. Popular benchmarks in 2026 include MMLU-Pro for broad knowledge, HumanEval and SWE-bench for code, MT-Bench for multi-turn conversation, Chatbot Arena for human preference, and TruthfulQA for factual accuracy. But benchmarks have blind spots. First, they cannot tell you how the model handles your specific use case. MMLU tests multiple-choice knowledge; your product might need long-form summarization, and the skills do not transfer one to one. Second, they cannot account for data contamination. If benchmark questions appeared in the model's training data, the score reflects memorization, not capability.

A **benchmark** is a fixed task set with a standardized scoring protocol that enables reproducible, apples to apples comparison of models across labs and time. The value is comparability. Because every model sees the same inputs and is scored the same way, a benchmark number is meaningful across contexts. MMLU-Pro tests broad multi-domain knowledge with 14,000 questions. HumanEval tests code generation on 164 programming problems. Chatbot Arena collects live human preferences through pairwise voting. Each benchmark carves out one dimension of capability and measures it precisely. Two things a benchmark cannot tell you. First, how the model will perform on your production workload. Your queries have a specific distribution, domain vocabulary, and difficulty profile that no general benchmark replicates. A model that tops MMLU-Pro may struggle on your legal document summarization task because the overlap between the benchmark and your data is small. Second, whether the score reflects genuine capability or contamination. If benchmark questions leaked into the training corpus, the model is recognizing answers from memory rather than reasoning about them. LiveBench and WildBench address this by using fresh, time-limited questions, but older fixed benchmarks remain vulnerable. The practical takeaway: use benchmarks for shortlisting models, then run your own task-specific eval to make the actual shipping decision.