When eval prompts appear verbatim inside an SFT file after a synthetic-data run, what's the root cause?

Attempt it

A team runs a synthetic-data generation pass over a labelled dataset to grow it 10x for SFT. After shipping the fine-tune, the on-call notices that several prompts from the eval set appear verbatim inside the training file. The generator was a strong teacher model and the input seeds were sampled from the original labelled pool. What specifically went wrong, and what is the correct process for splitting and generating?

Free · 2 AI evals / day

Quick reference

TL;DR

Eval seeds were fed into the synthetic generator, so paraphrases of held-out items landed in training and inflated the post-FT scores.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine writing a practice test for yourself by paying a tutor to invent new questions. You hand the tutor your real exam paper so they know the style. The tutor obligingly rewords each real question into ten new ones and you train on every one of them. Come exam day you ace it, but the score is fake because you essentially studied the answers. The fix is obvious once you see it. Keep the real exam locked in a drawer before the tutor sees anything. Only show the tutor a separate batch of warm-up problems. Then, after the tutor produces new questions, scan them against the real exam and throw out any that look suspiciously similar. The exam stays clean and your score reflects what you actually learned.

Key concepts

Data leakage Synthetic data generation Held-out evaluation Benchmark contamination Supervised fine-tuning

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

6 min: name seed-set contamination, walk through how paraphrases bypass string dedup, prescribe split-first-generate-second with embedding-level similarity checks, and add an external held-out shard.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's Evals and Anthropic's evaluation guidelines both recommend pre-registered eval splits that pre-date any synthetic augmentation pass.
Stanford's Alpaca and the LIMA paper from Meta both highlighted that synthetic SFT data inherits the seed distribution, motivating careful seed selection.
Hugging Face's datasets library exposes deterministic split helpers precisely so eval shards can be pinned before downstream augmentation.
Modern flagships like Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro publish data-decontamination procedures that scan benchmark items against training corpora using embedding similarity.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat similarity threshold and embedding model would you choose for the post-generation dedup scan?

Sign in or go Pro to see the approach for this follow-up.

QHow would you detect that a leak has already shipped in a deployed model?

Sign in or go Pro to see the approach for this follow-up.

QIf two teams share an upstream labelled pool, how do you keep their eval shards independent?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Splitting train and eval AFTER synthetic generation. By then the synthetic shard already encodes paraphrases of eval items and the leak is baked in.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.