Which of these are real train/test leakage modes for an instruction-tuning project?

Leakage means information secretly crosses from your eval set into something the model already saw, so the score looks better than the real skill. The test is simple: does the option move information across the train and eval boundary? Paraphrased duplicates count. If the same task appears in training worded one way and in the held-out set worded another way, the model has effectively been trained on the test. Exact-match dedup misses this because the strings differ. Pretraining contamination counts too. Public benchmarks like MMLU or GSM8K often sit inside the base model's pretraining data, so the model answers from memory, not reasoning. Shared-document chunks count. If two pieces of one document land on opposite sides of the split, document-specific knowledge leaks across. The two trap answers do not. File format like JSONL versus Parquet is just storage. Shuffling the training set is standard practice and never touches the eval set. Both sound technical but move zero information between splits.

Leakage is any path that lets the held-out set share information with training or with the base model's pretraining. The four real modes here are paraphrased duplicates, benchmark contamination, shared-document chunks, and teacher equals judge. Paraphrase duplicates are the classic subtle leak. The underlying task is identical across splits but the wording differs. String-level dedup passes while the model has effectively memorised the answer. Embedding-based near-duplicate detection is the fix. Benchmark contamination is leakage into pretraining. If your held-out benchmark already lived in the base corpus, both base and fine-tuned scores inflate. The fine-tuning delta you report is distorted as a result. The teacher-judge overlap is a sneakier mode. When the model that generated your training data shares a family with your evaluation judge, the judge rewards its own house style. The score rises without real quality moving. The two decoys are storage format and shuffle setting. Neither moves information across the boundary. JSONL versus Parquet is a serialisation choice. Shuffling reorders training batches but never pulls eval examples into training.