Which of these are real train/test leakage modes for an instruction-tuning project?

Attempt it

Quick reference

TL;DR

Leakage is any path that lets train and eval secretly share information. Paraphrase duplicates, pretraining contamination, shared-document chunks, and teacher equals judge all qualify; file format and shuffle do not.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a teacher grading a take-home exam. Leakage is anything that lets a student see the answer key before grading. If the practice sheet and the real exam reword the same question, the student looks brilliant but learned nothing. If the exam was already printed in the textbook everyone read, scores soar for free. If the same person who wrote the practice answers also grades the exam, they reward their own style. All three inflate the score without real skill. But the color of the paper, or the order pages were stapled, changes nothing. Those are the trick choices: they sound technical, yet they never move answers from one place to another.

Key concepts

Train/test leakage Benchmark contamination Near-duplicate detection LLM-as-judge bias

Quick scan · 2–4 min

Everything important, quickly.

5 min: the boundary-crossing test + paraphrase duplicates + pretraining contamination + shared-document chunks + teacher equals judge + why format and shuffle are decoys.

Side-by-side

Option	Real leakage?	Why
Paraphrased duplicates across splits	Yes	Same task, different wording; defeats exact-match dedup
Benchmark in base pretraining	Yes	Leak into pretraining; inflates the reported delta
Teacher family equals judge family	Yes	Judge rewards its own house style; eval-side bias
Same JSONL file format	No	Storage choice; moves no information across splits
Shared-document chunks across splits	Yes	Document-specific knowledge crosses the boundary
shuffle=True on the training set	No	Reorders batches; eval set is never trained on

Real-world examples

Real products, models, and research that use this idea.

The GSM8K and MMLU contamination findings showed many open base models had memorised public benchmark items, inflating reported scores before any fine-tuning.
Self-instruct and Alpaca-style pipelines generate training data with one model then judge with the same family, a teacher equals judge bias seen across instruction-tuning work.
OpenAI and Anthropic both publish decontamination procedures that strip benchmark overlaps from training data before reporting eval numbers for GPT-5.5 and Claude Opus 4.7.
RAGAS and LangSmith eval suites flag near-duplicate overlap between train and test splits using embedding similarity rather than exact string match.
Meta's Llama 4 technical reporting includes contamination analysis against GSM8K, HumanEval, and MMLU to separate genuine gains from memorisation.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you detect paraphrased duplicates that exact-match dedup misses?

Sign in or go Pro to see the approach for this follow-up.

QWhy does benchmark contamination distort the fine-tuning delta rather than just the absolute score?

Sign in or go Pro to see the approach for this follow-up.

QHow do you break the teacher equals judge bias in an LLM-as-judge eval?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.