Click any words you think contain an error. Click again to unmark.
Two errors: competition math can leak into pretraining, and reusing the same verifier in RL training and eval lets policies game the checker — inflating scores without true generalization.
Imagine a student who saw the exam questions online last year and practiced until the auto-grader gave full marks — even when their work skips real steps. High score, shaky skill. Reasoning benchmarks face the same traps: questions can leak into training data, and if RL trains against the exact same checker used for the leaderboard, the model learns the grader's quirks, not harder math.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Reasoning-model announcements cite AIME and MATH jumps as evidence of genuine progress. Skeptical interviewers — rightly — ask whether those numbers survive contamination audits and whether RL training gamed the same checker used for reporting. The spot-error passage states the naive optimistic view twice.
This card locates both false spans and builds the eval hygiene vocabulary senior candidates need.
Error 1: pretraining contamination is not zero
The passage claims competition problems never appear in pretraining corpora. Web-scale datasets scrape forums, solution PDFs, GitHub repos, and competition archives. AIME and MATH-style items propagate widely — sometimes verbatim, sometimes paraphrased in solution threads.
Contamination does not mean intentional cheating; it means the model may have seen similar problems during PT or mid-training, inflating few-shot or zero-shot benchmark performance without demonstrating fresh reasoning on truly novel tasks.
Practices that reduce false confidence:
- N-gram and embedding overlap filters against benchmark prompts
- Held-out private problem sets not published online
- Reporting performance only on decontaminated subsets
- Canary problems inserted to detect memorization signatures
Correction for chars 73-129: benchmark contamination from pretraining overlap is a known risk requiring holdout hygiene.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Multiple labs publish decontamination reports before claiming MATH/AIME improvements on frontier models.
- DeepSeek-R1 and peers emphasize verifiable RL while eval discussions note holdout and fresh problem needs.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do decontamination pipelines detect benchmark overlap in PT?
N-gram overlap thresholds, embedding NN search, manual audits on suspicious clusters.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating public math leaderboard jumps as pure reasoning gains — contamination and verifier leakage both inflate scores.
60 second bullets to scan on the way to the call.
State contamination risk for public competition math
Explain verifier leakage when RL reward equals eval scorer
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.