A distilled model hits 92% on a public benchmark but 31% on an internal hold out: diagnose.

Short answer·Medium·4.0 · 0·~3 min·Asked atAi21 Bain Banana Dev·Relevant atAnthropic Databricks OpenAI

Attempt it

A team distills a student model by SFT on synthetic answers generated by a strong frontier teacher. The student scores 92% on a widely cited public benchmark for the task and only 31% on an internal hold out drawn from the same real world task distribution. What is the most likely diagnosis, what mechanism produces this exact gap, and which eval do you trust to guide further work?

Free · 2 AI evals / day

The most likely diagnosis is benchmark contamination flowing through the teacher into the student. The student was trained on synthetic answers generated by the teacher model, not on the raw benchmark. So any benchmark-specific information in the student had to travel through the teacher first. The widely-cited public benchmark was almost certainly part of the teacher's pretraining or post-training corpus, because modern frontier models are trained on massive web corpora that include most popular benchmarks. When the teacher generates synthetic answers for distillation, it does not solve each prompt from first principles. It reproduces patterns, phrasings, and even answer keys it has memorized from its training corpus. The student then learns those patterns wholesale. This explains the exact gap. On the contaminated public benchmark, the student matches the teacher's leaked patterns, which are exactly what the benchmark scorer rewards. The score is artificially inflated. On the internal hold-out, those patterns do not apply because the hold-out was built inside the company and the teacher could not have seen it. The student is left with whatever genuine task ability it actually distilled, which is the honest 31 percent. The trustworthy eval is the internal hold-out. Treat the 92 percent as upper-bound noise. Steer further work using the hold-out as the primary metric, and consider contamination probes on the public benchmark to estimate how much of the 92 percent is real.

**Diagnosis: benchmark contamination flowing through the teacher into the student.** The student saw synthetic teacher outputs, not the raw benchmark. Any benchmark-specific signal had to come via the teacher. The widely-cited public benchmark was almost certainly in the teacher's training corpus, because frontier-model corpora include most well-known benchmarks. **The mechanism.** When the teacher generates synthetic answers for distillation, it draws on memory of the benchmark: specific question wordings, canonical phrasings, sometimes verbatim answer keys. The student inherits those patterns through SFT, mimicking outputs that are themselves contaminated. **Why the gap is exactly this shape.** On the contaminated public benchmark, the student's mimicry aligns with the scoring rubric, inflating the score to 92 percent. On the internal hold-out, the teacher never saw the questions, so the synthetic answers reflect only general task ability. The 31 percent is the honest measure. **Which eval to trust.** The internal hold-out. Always. Public benchmarks on a distilled model are upper-bound noise, not a steering signal. Concrete recipe: - Keep a hold-out the teacher could not have seen (after its training cutoff, internal-only). - Score against the hold-out as the primary metric. - Run contamination probes (canary strings, verbatim search, temperature-zero stability) to estimate leakage. - Filter teacher outputs that look memorized (high reproducibility, exact match to known keys). Distillation inherits the teacher's memorized advantages. Eval methodology has to account for both.

TL;DR

Benchmark contamination via the teacher. The public benchmark was in the teacher's training corpus, so the teacher's synthetic answers leak benchmark-specific patterns the student inherits. Trust the internal hold-out.

Memory aid

Easy to grasp

Picture hiring a tutor to write practice answers for your student, then grading the student on two tests. One test is a famous textbook the tutor has already read and memorized. The other test is a custom set of questions your school wrote internally. The tutor's practice answers are quietly shaped by their memory of the famous textbook, so when the student takes that test, the answers line up surprisingly well. On the custom test the tutor never saw, the practice answers do not carry the same hidden help, and the student scores closer to their real ability. The honest grade is the custom test. The famous-textbook grade is inflated by the tutor's memory leaking through.

Key concepts

Distillation Evaluation methodology Benchmark contamination Internal hold-outs

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The signature of distillation contamination is the public-versus-internal gap. A model that scores nearly perfectly on a famous public benchmark and falls apart on an internal hold-out drawn from the same task distribution is almost never failing because of model capacity, training schedule, or LoRA rank. It is failing because the public benchmark was in the teacher's training corpus, the teacher's synthetic answers carried memorized patterns into the student, and the internal hold-out exposes the gap between memorized mimicry and genuine skill.

This pattern is increasingly common as distillation becomes a standard recipe for building cheaper task-specific models on top of frontier teachers. The teachers (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro, Llama 4 Maverick, DeepSeek V4) all trained on web-scale corpora that include nearly every popular benchmark, often unfiltered. Distilled students inherit the teacher's memorized advantages and pay the same penalty on truly unseen evaluations.

This deep dive walks through the data lineage that produces the gap, the precise mechanism that makes the 92-versus-31 split the contamination signature, the construction principles for a trustworthy hold-out, the contamination probes that quantify the leak, and the broader implications for evaluation methodology on distilled models.

The data lineage that produces the gap

Follow the data backward from the student's eval score to the source of the signal.

The student was trained on SFT data consisting of synthetic answers from the teacher. The student never saw the public benchmark directly. So any benchmark-specific signal in the student had to be transmitted through the teacher's outputs.

The teacher's outputs are themselves a function of the teacher's training corpus. If the teacher had the public benchmark in its training corpus, two things happen when the teacher is asked to generate an answer to a prompt that resembles a benchmark question. First, the teacher draws on its memory of the canonical benchmark answer (specific wordings, formatting conventions, even verbatim text in some cases). Second, the teacher's general task ability is also engaged. The synthetic output is a mixture of memorized leakage and genuine skill.

The student inherits both components through SFT. On the contaminated benchmark, the leaked-pattern component pays off because the patterns were learned from that benchmark's expected answers. On the internal hold-out, only the genuine-skill component pays off, because the leaked patterns are specific to the benchmark and do not transfer.

The lineage explains both the inflation on the public benchmark and the honest score on the hold-out. The size of the gap is roughly the size of the contamination contribution to the teacher's outputs.

Why the 92-versus-31 split is the contamination signature

Constructing a trustworthy hold-out

Contamination probes that quantify the leak

Implications for distillation methodology

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Eval type	What it measures (on a distilled model)	Trust level
Widely-cited public benchmark	Leaked patterns + genuine skill (entangled)	Low; use only as upper bound
Internal hold-out (teacher cannot have seen)	Genuine distilled skill on the task distribution	High; primary steering metric
Held-out slice of public benchmark	Slightly less leakage, still suspect	Medium; useful for trend, not absolute
Adversarial probes (canary strings, etc.)	Quantifies the leak directly	Use as diagnostic, not as headline metric

Real products, models, and research that use this idea.

Benchmark contamination concerns have prompted leaderboard maintainers like LiveCodeBench and SciCode to construct test sets with explicit post-publication cutoffs, designed to be unseen by major frontier models.
Practitioners distilling from Claude Opus 4.7, GPT-5.5, or Gemini 3.1 Pro routinely keep internal hold-outs constructed from proprietary data precisely to surface contamination flowing through synthetic answers.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you construct an internal hold-out that you can be confident the teacher never saw?

Pick a construction date after the teacher's training-data cutoff. Source from internal-only data (customer interactions under NDA, proprietary documents, post-cutoff news). Avoid drawing from public datasets even with paraphrasing. Validate by attempting verbatim search of sample questions against the teacher's known training sources where possible.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating the 92% on the public benchmark as evidence the distillation worked. The teacher likely had the benchmark in its training corpus, and the student inherited the leaked patterns through the synthetic answers.

60 second bullets to scan on the way to the call.

Why distillation transmits not just capability but also memorized patterns
How contamination in the teacher flows through synthetic answers into the student

Primary sources. Browse if you want the original framing.