Spot the flaw: 'We fine-tuned on 10K examples and eval on a random 500 from the same dataset. Accuracy is 94%, so the model is ready for production.'

The flaw is **data leakage** between training and evaluation. The 500 eval examples came from the same 10,000 example dataset used for fine-tuning. Some or all of those 500 examples may have been in the training set, so the model might be recalling memorized answers rather than demonstrating genuine understanding. Even if by luck the 500 examples have zero overlap with the training set, both groups share the same collection methodology, annotator pool, and distribution characteristics. Production traffic will look different from your curated dataset, so the 94% number overestimates real world performance. The fix is to split before training, not after. Hold out a test set that never touches the training pipeline. Better still, collect the eval set from a different source entirely, like sampled production traffic from a different time window. Then run a contamination check to verify zero exact overlap between training and eval as a hard gate before reporting results.

The flaw is **data leakage**: the eval set was drawn from the same pool as the training data, so the 94% accuracy conflates memorization with generalization. Two layers of the problem. First, any of the 500 eval examples that also appeared in the 10K training set are testing recall, not capability. A model that memorized all 10K examples would score perfectly on any subset drawn from them. Second, even without exact overlap, both sets share collection bias: same annotators, same time window, same distribution of edge cases. Production traffic will not match this distribution, so accuracy drops when the model deploys. The fix has three parts. Split the data before training and hold out the eval set permanently. Collect eval data independently if possible, using production traffic or a separate annotation pass. Add a contamination check as a pipeline gate: hash every example and verify zero overlap between train and eval before reporting any metric.