Explain benchmark contamination and its effect on reported LLM capability scores

**Benchmark contamination** happens when the questions and answers from an evaluation set leak into the model's pre-training data. The model effectively studied the test before it sat the exam. When this happens, a high score does not mean the model is smart. It means the model memorized the answer and is reciting it. The difference matters: memorization is recall, real capability is solving a problem the model has never seen. A contaminated benchmark hides that distinction. The practical effect is score inflation. A leaderboard might show 90 percent on a math benchmark, but if a chunk of that benchmark was in the training corpus, the true reasoning ability could be far lower. You are misled about how good the model actually is. Detection is hard for three reasons. Pre-training corpora are usually secret, so you cannot just search them. Paraphrased versions of questions slip past simple word-matching checks. And the model's weights do not store a receipt saying which examples it saw. So contamination is easy to suspect but hard to prove.

**Benchmark contamination** is when evaluation examples appear, verbatim or near-verbatim, in a model's pre-training corpus. The model then recalls the answer instead of deriving it. The mechanism is memorization versus generalization. A benchmark is meant to test reasoning on unseen problems. If the test items leaked into training, a high score reflects recall of the gold answer, not genuine skill. The reported number overstates real capability, and the gap is largest on the items that leaked. Detection after the fact is genuinely hard, for three reasons. Pre-training corpora are rarely disclosed, so you cannot directly search for overlap. Near-duplicate paraphrases evade n-gram and substring checks, since the surface text differs. And weights retain no explicit pointer back to training examples, so you cannot trace an output to its source. Practitioners therefore rely on indirect signals. A large performance drop on a paraphrased or freshly authored variant suggests memorization. So does strong performance only on benchmarks published before the training cutoff. The consequence is that leaderboard rankings can mislead model selection unless you guard with private held-out sets and fresh tasks.