Spot the flaw: 'We fine-tuned on 10K examples and eval on a random 500 from the same dataset. Accuracy is 94%, so the model is ready for production.'
Click any words you think contain an error. Click again to unmark.
The eval set was drawn from the same pool as the training data. The 94% accuracy may measure memorization, not generalization. Use a held out set collected independently.
Imagine a teacher gives students a practice test with 100 questions, then tests them on 10 questions randomly picked from the same 100. The students score well because they already practiced those exact questions. The teacher thinks the students understand the material, but they might just have good memories. A fair test would use new questions the students have never seen, drawn from a different source. That is what a held out eval set does for a model.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The scenario describes a textbook data leakage failure. The team fine-tuned on 10K examples and evaluated on 500 examples drawn from the same 10K. The 94% accuracy looks impressive until you realize the experimental design cannot distinguish between a model that learned the task and a model that memorized its training data.
This is one of the most common evaluation mistakes in applied ML. It ships because the workflow feels reasonable: you have labeled data, you train on it, you test on a sample from it, and the numbers look good. The problem is invisible until the model meets production traffic and accuracy drops sharply. The rest of this explanation walks through why this happens, how to fix it, and how to catch it automatically.
Why train eval overlap invalidates accuracy
When a model is fine-tuned on a dataset and then evaluated on examples from the same dataset, the eval measures a mixture of generalization and memorization. The model may have learned the underlying task, or it may have memorized the specific examples it was trained on, or some combination of both. The eval cannot tell you which.
For the specific numbers in the scenario: 500 examples sampled randomly from 10K. Without excluding the training set, the expected overlap depends on the sampling method. If sampling with replacement from the full 10K, each eval example has a 100% chance of being in the training set (since training used all 10K). The eval is literally testing memorization.
A model with perfect memorization of its training data would score 100% on this eval regardless of whether it learned anything about the task. The 94% accuracy tells you the model memorized at least 94% of the examples it saw during training, plus whatever it genuinely generalized. You cannot separate the two signals.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Kaggle competitions enforce strict train test splits with hidden test sets specifically to prevent this class of leakage, and contestants who accidentally train on test data are disqualified.
- The Alpaca Eval and LMSYS Chatbot Arena leaderboards have documented cases where models scored higher on benchmarks contaminated into their training data, leading to contamination detection becoming a standard part of evaluation.
What an interviewer would ask next. Try answering before peeking at the approach.
QYou verified zero exact overlap between train and eval, but the eval accuracy is still suspiciously high. What else could cause inflated scores?
Near duplicate contamination: paraphrased or lightly edited versions of training examples in the eval set. Distribution overlap: both sets share the same easy examples and the same annotator biases. To catch near duplicates, compute embedding similarity between all train eval pairs and flag pairs above a cosine threshold (typically 0.95). To catch distribution overlap, compare the eval set distribution against production traffic.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Drawing the eval set from the same pool as training data and interpreting high accuracy as evidence of generalization rather than memorization.
60 second bullets to scan on the way to the call.
Why sampling eval data from the training pool invalidates the accuracy metric.
The difference between exact overlap and distribution leakage.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.