A model aces a benchmark with suspiciously high scores. The first hypothesis is contamination. Explain what that means to a non-technical PM.

**Contamination** is what happens when a model's training data includes the benchmark questions it will later be tested on. The model memorizes the specific answers rather than learning to reason about the underlying concepts. The result is a benchmark score that looks impressive but does not reflect real capability. The model aces questions it has seen before but may struggle on novel questions that test the same skills in a different way. The analogy for a PM: imagine a job candidate who somehow got the interview questions in advance and memorized perfect answers. They sound brilliant in the interview, but on their first day they cannot handle a problem they did not rehearse. The interview score was real, but the capability it was supposed to measure was not. Contamination happens because LLMs are trained on massive internet corpora, and benchmark datasets are often published online. MMLU questions, HumanEval problems, and TruthfulQA prompts are all available on the web. If they end up in the training set, the model has effectively studied the answer key. This is why benchmark designers use held-out sets, canary strings, and temporal cutoffs to detect and prevent contamination. It is also why the field increasingly values live evaluations like Chatbot Arena, where new prompts come from real users and cannot be memorized in advance.

To a PM, here is the explanation. **Contamination** means the model saw the test questions during training and memorized the answers. The benchmark score is inflated because the model is recalling, not reasoning. Think of it as the difference between understanding math and memorizing the answer key. Both produce a perfect score, but only one generalizes to new problems. A contaminated benchmark tells you nothing about how the model will perform on real user queries it has never seen. Contamination happens because LLMs train on enormous internet datasets. Benchmark questions are often published online (MMLU on GitHub, HumanEval in papers, TruthfulQA in research repos). If the training pipeline does not filter them out, the model absorbs the exact questions and answers. Detection methods include checking for suspiciously high scores on specific benchmark subsets, looking for memorized verbatim sequences, and testing on rephrased versions of the same questions. If the score drops sharply on rephrased versions, contamination is likely. The practical consequence: do not trust a single benchmark score in isolation. Cross-reference with live evaluations (Chatbot Arena), freshly constructed private test sets, and rephrased variants. If the model only excels on the exact published questions, the score is measuring memorization, not capability.