Every model release cites benchmark numbers. Define what a benchmark is and name two things it cannot tell you.

A **benchmark** is a standardized evaluation: a fixed set of tasks with a predetermined scoring method that lets anyone compare model performance on equal terms. The key word is standardized. Everyone uses the same inputs, the same scoring function, and the same split between test and development data. This makes scores directly comparable across labs, model versions, and time. When two labs report MMLU-Pro scores for their models, you know both numbers were computed on the same test. Popular benchmarks include MMLU-Pro for broad knowledge, HumanEval and SWE-bench for code, MT-Bench for multi-turn conversation, Arena for human preference, and TruthfulQA for factual accuracy. But benchmarks have blind spots. First, they cannot tell you how the model handles your specific use case. MMLU tests multiple-choice knowledge. Your product might need long-form summarization, and the skills do not transfer one to one. Second, they cannot account for data contamination. If benchmark questions appeared in the model's training data, the score reflects memorization, not capability.

A **benchmark** is a fixed task set with a standardized scoring protocol that enables reproducible, direct comparison of models across labs and time. The value is comparability. Because every model sees the same inputs and is scored the same way, a benchmark number is meaningful across contexts. MMLU-Pro tests broad multi-domain knowledge with 12,032 questions. HumanEval tests code generation on 164 programming problems. Arena collects live human preferences through pairwise voting. Each benchmark isolates one dimension of capability and measures it precisely. A benchmark cannot tell you two things. First, it cannot tell you how the model will perform on your production workload. Your queries have a specific distribution, domain vocabulary, and difficulty profile that no general benchmark replicates. A model that tops MMLU-Pro may struggle on your legal document summarization task because the overlap between the benchmark and your data is small. Second, it cannot tell you whether the score reflects genuine capability or contamination. If benchmark questions leaked into the training corpus, the model is recognizing answers from memory rather than reasoning about them. LiveBench addresses this with fresh, time-limited questions, and WildBench uses real user tasks collected from chatbot logs. Older fixed benchmarks remain vulnerable. In practice, use benchmarks for shortlisting models, then run your own task-specific eval to make the actual shipping decision.