N-gram overlap, canary strings, paraphrase accuracy drop, and the perplexity gap all detect benchmark contamination. Weight pruning does not, and temperature-1 sampling only catches the verbatim case.
Imagine a student who somehow saw the exam paper before the test. How do you catch them? First, you compare their answers word for word against the leaked sheet (n-gram overlap). Second, you slip a made-up nonsense fact into the textbook only they had, then ask about it; if they know it, they read that exact copy (canary string). Third, you reword the questions and watch their score collapse, which means they memorized phrasing not ideas (paraphrase test). Fourth, you notice they answer the leaked questions with eerie, instant confidence compared to fresh ones (low surprise, the perplexity gap). What does not work: tearing the textbook apart page by page to find the leak (weight pruning), and simply asking the same question again, which only catches the clumsiest copying.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Benchmark contamination is the silent killer of LLM evaluation. When eval questions or answers leak into the pretraining corpus, the model scores high by recalling memorized text rather than demonstrating the skill the benchmark claims to measure. Every downstream decision built on that number, from model selection to release gating, inherits the lie. By 2026, with frontier models trained on multi-trillion token web scrapes that absorb GitHub, Stack Exchange, and the very leaderboards we evaluate against, assuming a public benchmark is clean is naive rather than cautious.
The problem compounds because contamination is not binary. A benchmark can leak verbatim, in paraphrase, in translation, as a solution walkthrough on a blog, or as a discussion thread that reveals the answer without restating the question. Each route leaves a different fingerprint, and no single detector sees all of them. That is why a credible audit layers multiple methods rather than trusting one.
This question lists six candidate detection methods. Four are practical and widely used; two are distractors. The clean way to organize the valid four is by what access they require: corpus-side methods need the training data, behavioral methods need only the model. The deep dive walks each method, its signal, its blind spot, and the real audits that have codified it.
N-gram overlap: the corpus-side workhorse
The most direct contamination test is also the oldest. Take each eval example, slice it into n-grams (commonly 8-gram to 13-gram spans), and search the training corpus for exact matches. If a long, distinctive span from your test set appears verbatim in pretraining, that example is contaminated. The GPT-3 and GPT-4 technical reports both ran this and reported decontaminated scores alongside the raw ones.
The method is cheap, interpretable, and precise on exact text. Its blind spot is equally clear: it only catches surface-form matches. Any paraphrase, translation, reformatting, or change of solution order slips through, because the n-grams no longer line up. A benchmark answer that appears in training as a reworded forum post will pass an n-gram check while still contaminating the model.
N-gram overlap also requires read access to the pretraining corpus, which rules it out for closed third-party models. It is a lab-internal audit tool, run by the team that owns the data, not by an external evaluator probing an API.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Frontier model cards from labs like the Llama 4 and GPT-5.5 teams run substring n-gram contamination checks against the training corpus and report decontaminated scores.
- BIG-bench embeds a canary GUID string so model builders can detect and exclude the benchmark, and auditors can probe for its recall.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a contamination audit for a closed model you can only query through an API?
Lean on behavioral signals: paraphrase the benchmark and measure accuracy delta, then run perplexity or Min-K percent prob against difficulty-matched held-out text. Triangulate both, since either alone has false positives.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a clean public benchmark score proves capability. If the test set leaked into pretraining, the number measures memorization, not generalization, and every downstream decision built on it is wrong.
60 second bullets to scan on the way to the call.
N-gram overlap against the training corpus and its blind spots
Canary strings as provenance proof and their pre-training requirement
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.