Which of these are real train/test leakage modes for an instruction tuning project?
Leakage is any path that lets train and eval secretly share information. Paraphrase duplicates, pretraining contamination, shared-document chunks, and teacher equals judge all qualify; file format and shuffle do not.
Imagine a teacher grading a take-home exam. Leakage is anything that lets a student see the answer key before grading. If the practice sheet and the real exam reword the same question, the student looks brilliant but learned nothing. If the exam was already printed in the textbook everyone read, scores soar for free. If the same person who wrote the practice answers also grades the exam, they reward their own style. All three inflate the score without real skill. But the color of the paper, or the order pages were stapled, changes nothing. Those are the trick choices: they sound technical, yet they never move answers from one place to another.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Train/test leakage is any mechanism that lets the held-out evaluation share information with what the model has already learned, whether from fine-tuning data or from the base model's pretraining. When that happens, the eval score climbs without the underlying skill improving. The whole point of a held-out set is to estimate generalisation, and leakage quietly destroys that estimate.
The useful mental tool for this question is a single test: does the option create a channel for information to cross the boundary between train and eval, or between eval and pretraining? If yes, it is a leakage mode. If no, it is an engineering detail that has nothing to do with leakage, no matter how technical it sounds.
This question is built around that test. Four of the six options open a real channel, in four distinct ways. Two options are decoys. They describe legitimate engineering choices like file serialisation and batch shuffling that never relocate an example or its information from one side of the boundary to the other. The skill being probed is the ability to separate true leakage from plausible-sounding noise, which is exactly the judgment that keeps a fine-tuning eval honest.
The boundary-crossing test
Every genuine leakage mode shares one property. Information that should live only in the eval set ends up available to the model, either directly or through the grader. So the diagnostic is mechanical. Pick the option, then ask whether it relocates information across the train and eval line.
Apply it to the decoys first. Storing the eval set as JSONL rather than Parquet or CSV is a serialisation decision. The bytes on disk change; the contents of each split do not. No eval example becomes visible to training. Shuffling the training set reorders the batches the optimiser sees. It improves gradient estimates and convergence, but the eval set is held out regardless of order. Neither option moves a single example across the boundary, so neither is leakage.
Apply the same test to the four real modes and each one passes. A task reappears across splits, a benchmark sits in pretraining, document knowledge spans the split, or the judge has privileged knowledge of the teacher's style. The test is the whole answer.
Why lean on one diagnostic rather than memorising a list of leakage types? Because new variants keep appearing as pipelines grow more elaborate. Synthetic data generation, retrieval-augmented training, and multi-stage distillation all invent fresh ways for information to cross the line. A memorised taxonomy goes stale; the boundary-crossing question generalises. Whenever you add a stage to a pipeline, ask it of that stage, and you will catch leaks the textbook list never named.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Option | Real leakage? | Why |
|---|---|---|
| Paraphrased duplicates across splits | Yes | Same task, different wording; defeats exact-match dedup |
| Benchmark in base pretraining | Yes | Leak into pretraining; inflates the reported delta |
| Teacher family equals judge family | Yes | Judge rewards its own house style; eval-side bias |
| Same JSONL file format | No | Storage choice; moves no information across splits |
| Shared-document chunks across splits | Yes | Document-specific knowledge crosses the boundary |
| shuffle=True on the training set | No | Reorders batches; eval set is never trained on |
Real products, models, and research that use this idea.
- The GSM8K and MMLU contamination findings showed many open base models had memorised public benchmark items, inflating reported scores before any fine-tuning.
- Self-instruct and Alpaca-style pipelines generate training data with one model then judge with the same family, a teacher equals judge bias seen across instruction-tuning work.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect paraphrased duplicates that exact-match dedup misses?
Embed every example with a sentence encoder, then flag pairs above a cosine threshold across splits. Tune the threshold on a labelled sample and dedup at the cluster level before splitting.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Relying on exact-match dedup and assuming the splits are clean. Paraphrased duplicates and shared-document chunks slip straight through string matching and quietly inflate the eval.
60 second bullets to scan on the way to the call.
The boundary-crossing test for what counts as leakage
Why exact-match dedup misses paraphrased duplicates
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.