Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
The flaw is **data leakage** between training and evaluation. The 500 eval examples came from the same 10,000 example dataset used for fine-tuning. Some or all of those 500 examples may have been in the training set, so the model might be recalling memorized answers rather than demonstrating genuine understanding.
Even if by luck the 500 examples have zero overlap with the training set, both groups share the same collection methodology, annotator pool, and distribution characteristics. Production traffic will look different from your curated dataset, so the 94% number overestimates real world performance.
The fix is to split before training, not after. Hold out a test set that never touches the training pipeline. Better still, collect the eval set from a different source entirely, like sampled production traffic from a different time window. Then run a contamination check to verify zero exact overlap between training and eval as a hard gate before reporting results.
The flaw is **data leakage**: the eval set was drawn from the same pool as the training data, so the 94% accuracy conflates memorization with generalization.
Two layers of the problem. First, any of the 500 eval examples that also appeared in the 10K training set are testing recall, not capability. A model that memorized all 10K examples would score perfectly on any subset drawn from them. Second, even without exact overlap, both sets share collection bias: same annotators, same time window, same distribution of edge cases. Production traffic will not match this distribution, so accuracy drops when the model deploys.
The fix has three parts. Split the data before training and hold out the eval set permanently. Collect eval data independently if possible, using production traffic or a separate annotation pass. Add a contamination check as a pipeline gate: hash every example and verify zero overlap between train and eval before reporting any metric.
Quick reference
TL;DR
The eval set was drawn from the same pool as the training data. The 94% accuracy may measure memorization, not generalization. Use a held out set collected independently.
Imagine a teacher gives students a practice test with 100 questions, then tests them on 10 questions randomly picked from the same 100. The students score well because they already practiced those exact questions. The teacher thinks the students understand the material, but they might just have good memories. A fair test would use new questions the students have never seen, drawn from a different source. That is what a held out eval set does for a model.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Cross validation on a small dataset where you cannot afford to hold out data permanently. k-fold cross validation deliberately trains and evaluates on overlapping splits, but it averages across folds to estimate generalization. The scenario in the question is not cross validation; it is a single train eval split with no exclusion.
Few shot evaluation where the eval examples are part of the prompt, not the training data. In context learning puts examples in the prompt at inference time. The model has not been trained on them, so there is no memorization risk in the training sense.
A benchmark that is intentionally included in pre-training to measure ceiling performance. Some research setups deliberately contaminate to measure the gap between contaminated and clean performance. This is valid research methodology but not valid evaluation methodology.
Quick scan · 2–4 min
Everything important, quickly.
Open by naming the flaw: train eval contamination from drawing the eval set from the training pool. Walk the two layers: exact overlap (memorization) and distribution leakage (shared bias). Explain the fix: split before training, independent collection, contamination check as a pipeline gate. Cover the generalization gap metric. Close on why the 94% number is untrustworthy and what a trustworthy eval design looks like.
Real-world examples
Real products, models, and research that use this idea.
1Kaggle competitions enforce strict train test splits with hidden test sets specifically to prevent this class of leakage, and contestants who accidentally train on test data are disqualified.
2The Alpaca Eval and LMSYS Chatbot Arena leaderboards have documented cases where models scored higher on benchmarks contaminated into their training data, leading to contamination detection becoming a standard part of evaluation.
3Production ML teams at major tech companies run contamination detection pipelines that hash every training example and check against eval sets before reporting any metric, catching leakage that manual inspection misses.
4The Open LLM Leaderboard on Hugging Face added contamination detection after discovering that some submitted models had benchmark data in their training sets, inflating scores by 5 to 15 points.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYou verified zero exact overlap between train and eval, but the eval accuracy is still suspiciously high. What else could cause inflated scores?
Sign in or go Pro to see the approach for this follow-up.
QHow would you design a contamination detection pipeline that runs automatically in CI?
Sign in or go Pro to see the approach for this follow-up.
QYour eval set was collected from production traffic six months after training. Can you trust it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Drawing the eval set from the same pool as training data and interpreting high accuracy as evidence of generalization rather than memorization.