A team distills a student model by SFT on synthetic answers generated by a strong frontier teacher. The student scores 92% on a widely-cited public benchmark for the task and only 31% on an internal hold-out drawn from the same real-world task distribution. What is the most likely diagnosis, what mechanism produces this exact gap, and which eval do you trust to guide further work?
Free · 2 AI evals / day
The most likely diagnosis is benchmark contamination flowing through the teacher into the student.
The student was trained on synthetic answers generated by the teacher model, not on the raw benchmark. So any benchmark-specific information in the student had to travel through the teacher first. The widely-cited public benchmark was almost certainly part of the teacher's pretraining or post-training corpus, because modern frontier models are trained on massive web corpora that include most popular benchmarks.
When the teacher generates synthetic answers for distillation, it does not solve each prompt from first principles. It reproduces patterns, phrasings, and even answer keys it has memorized from its training corpus. The student then learns those patterns wholesale.
This explains the exact gap. On the contaminated public benchmark, the student matches the teacher's leaked patterns, which are exactly what the benchmark scorer rewards. The score is artificially inflated. On the internal hold-out, those patterns do not apply because the hold-out was built inside the company and the teacher could not have seen it. The student is left with whatever genuine task ability it actually distilled, which is the honest 31 percent.
The trustworthy eval is the internal hold-out. Treat the 92 percent as upper-bound noise. Steer further work using the hold-out as the primary metric, and consider contamination probes on the public benchmark to estimate how much of the 92 percent is real.
**Diagnosis: benchmark contamination flowing through the teacher into the student.**
The student saw synthetic teacher outputs, not the raw benchmark. Any benchmark-specific signal had to come via the teacher. The widely-cited public benchmark was almost certainly in the teacher's training corpus, because frontier-model corpora include most well-known benchmarks.
**The mechanism.** When the teacher generates synthetic answers for distillation, it draws on memory of the benchmark: specific question wordings, canonical phrasings, sometimes verbatim answer keys. The student inherits those patterns through SFT, mimicking outputs that are themselves contaminated.
**Why the gap is exactly this shape.** On the contaminated public benchmark, the student's mimicry aligns with the scoring rubric, inflating the score to 92 percent. On the internal hold-out, the teacher never saw the questions, so the synthetic answers reflect only general task ability. The 31 percent is the honest measure.
**Which eval to trust.** The internal hold-out. Always. Public benchmarks on a distilled model are upper-bound noise, not a steering signal. Concrete recipe:
- Keep a hold-out the teacher could not have seen (after its training cutoff, internal-only).
- Score against the hold-out as the primary metric.
- Run contamination probes (canary strings, verbatim search, temperature-zero stability) to estimate leakage.
- Filter teacher outputs that look memorized (high reproducibility, exact match to known keys).
Distillation inherits the teacher's memorized advantages. Eval methodology has to account for both.
Quick reference
TL;DR
Benchmark contamination via the teacher. The public benchmark was in the teacher's training corpus, so the teacher's synthetic answers leak benchmark-specific patterns the student inherits. Trust the internal hold-out.
Picture hiring a tutor to write practice answers for your student, then grading the student on two tests. One test is a famous textbook the tutor has already read and memorized. The other test is a custom set of questions your school wrote internally. The tutor's practice answers are quietly shaped by their memory of the famous textbook, so when the student takes that test, the answers line up surprisingly well. On the custom test the tutor never saw, the practice answers do not carry the same hidden help, and the student scores closer to their real ability. The honest grade is the custom test. The famous-textbook grade is inflated by the tutor's memory leaking through.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Distilling from a teacher whose training corpus is provably free of the evaluation benchmark Without contamination, the public-versus-hold-out gap reflects genuine difficulty differences, not leakage; the diagnosis here does not apply.
Evaluating a frontier model on the benchmark directly (not a distilled student) Direct evaluation has its own contamination story, but it is not transmitted through a teacher; analyze the model's own training-data exposure instead.
The internal hold-out has been used heavily during iteration and may itself be leaked Hold-outs that leak into the iteration loop lose their honesty; build a fresh held-out for new measurements.
The public benchmark is recently constructed with strict post-publication evaluation protocols Newer benchmarks designed against contamination (LiveCodeBench-style, SciCode-style) may be cleaner; verify their construction protocol before discarding their scores.
Quick scan · 2–4 min
Everything important, quickly.
5 min: state the diagnosis, trace the data lineage from teacher to student, explain why the exact 92 versus 31 split is the contamination signature, then prescribe the hold-out as the steering eval.
Side-by-side
Eval type
What it measures (on a distilled model)
Trust level
Widely-cited public benchmark
Leaked patterns + genuine skill (entangled)
Low; use only as upper bound
Internal hold-out (teacher cannot have seen)
Genuine distilled skill on the task distribution
High; primary steering metric
Held-out slice of public benchmark
Slightly less leakage, still suspect
Medium; useful for trend, not absolute
Adversarial probes (canary strings, etc.)
Quantifies the leak directly
Use as diagnostic, not as headline metric
Real-world examples
Real products, models, and research that use this idea.
1Benchmark contamination concerns have prompted leaderboard maintainers like LiveCodeBench and SciCode to construct test sets with explicit post-publication cutoffs, designed to be unseen by major frontier models.
2Practitioners distilling from Claude Opus 4.7, GPT-5.5, or Gemini 3.1 Pro routinely keep internal hold-outs constructed from proprietary data precisely to surface contamination flowing through synthetic answers.
3The 2024-2026 wave of contamination studies (canary-string probes, training-data overlap analysis) has documented public benchmarks being present in frontier-model corpora across nearly every popular evaluation.
4Open-source distillation projects like Zephyr and OpenHermes report public-benchmark gaps that often shrink when held-out evaluations are added, reflecting the same leakage pattern this question describes.
5Production model vendors increasingly publish dual scores (public benchmark plus held-out task) on distilled models, acknowledging that the public score alone is no longer a credible signal.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you construct an internal hold-out that you can be confident the teacher never saw?
Sign in or go Pro to see the approach for this follow-up.
QWhat contamination probe would you run to quantify how much of the 92 percent on the public benchmark is leaked?
Sign in or go Pro to see the approach for this follow-up.
QHow would you communicate model quality to stakeholders when the public benchmark score is inflated?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the 92% on the public benchmark as evidence the distillation worked. The teacher likely had the benchmark in its training corpus, and the student inherited the leaked patterns through the synthetic answers.