When eval prompts appear verbatim inside an SFT file after a synthetic-data run, what's the root cause?

**Data leakage** is when something from the eval set sneaks into training, so the eval score measures memory rather than skill. Here the leak came through the synthetic-data step. The team had one labelled pool. They fed examples from that pool into a strong teacher model and asked it to generate paraphrases. Then they trained on all the paraphrases and used a held-out slice as eval. The problem is that the held-out slice was never carved out before the teacher saw the data. So eval items, or close paraphrases of them, ended up in training. The symptom is high eval numbers that do not survive real use. The model is partly being scored on questions it already studied. The fix is procedural. Split first, generate second. Cut the eval shard out before anything else touches the pool. Only feed train-shard examples to the teacher. After generation, run a similarity check that compares each synthetic example against the eval shard and drops near-duplicates. From the moment the eval shard exists, treat it as untouchable.

**The root cause is seed-set contamination.** The synthetic generator was prompted with seeds drawn from the same labelled pool that the eval shard was later carved out of. A strong teacher tends to produce close paraphrases of the seed it was shown, and the dedup step compared strings rather than meaning, so the paraphrases passed through. The training file now contains many examples whose semantic content matches eval items, and the post-FT eval numbers reflect that overlap rather than true generalisation. The correct process is split first, generate second. Cut a fixed eval shard out of the labelled pool before any synthetic work runs. Only train-shard examples become seeds for the teacher. After the synthetic pass, run a similarity check between each new example and every eval item, then drop pairs above a chosen threshold. Embedding cosine similarity catches most paraphrases that exact-string dedup misses. A stricter shop runs two eval shards. The first is held out before any synthetic work and protects against pipeline leakage. The second comes from a different source entirely and protects against the labelled pool itself being upstream of the model under test. Both are write-once and ride alongside the model checkpoint so future runs can re-verify.