In classical ML the data splits into train/val/test. For LLM evaluation, which split matters most and why?

In classical machine learning, you split your data into three parts. The training set teaches the model, the validation set helps you tune hyperparameters and catch overfitting, and the test set gives you a final unbiased performance estimate. **LLM evaluation changes this picture.** When you use a model through an API (Claude, GPT, Gemini), you did not train it and you do not control its validation. The model arrives pre-trained and instruction-tuned by the provider. The only split you fully own is the test set, the held-out eval set you use to measure whether the model meets your quality bar. This makes the test set the most important artifact in your evaluation practice. Its coverage of real production queries determines whether your eval results generalize. Its contamination status (whether the model already saw these examples during training) determines whether your scores are inflated. And its alignment with actual user behavior determines whether high test scores translate to happy users. The practical implication: spend your evaluation budget on building, curating, and maintaining a high-quality test set rather than worrying about train/val splits that you do not control.

In classical ML the three-way split serves three purposes: train teaches the model, val tunes hyperparameters, and test gives a final performance estimate. For **LLM evaluation**, the picture collapses. Most LLM deployments consume the model as an API. The application team does not run training (no train set) and does not tune hyperparameters against a validation loss (no val set). The only split the team fully controls is the test set, the held-out eval set used to measure model quality on the tasks that matter. Three properties of the test set determine whether evaluation results are trustworthy. Coverage: does the set represent the real distribution of production queries, including edge cases? Contamination status: has the model already seen these examples during pre-training, which would inflate scores and hide weaknesses? Alignment: do high test scores actually predict user satisfaction, or is the test set testing the wrong things? This is why LLM evaluation is sometimes called 'all about the test set.' The model is a black box you cannot retrain. The prompt is the primary tuning knob. And the test set is the measurement instrument that tells you whether your prompt changes helped or hurt. If the instrument is bad, every decision downstream is unreliable.