Which buckets MUST a post-FT eval suite include for a customer-support fine-tune?

When you **fine-tune** a model, the dataset you trained on is NOT how you judge whether it worked. You need a separate eval suite that tests behaviour the model has never seen. For a customer-support fine-tune, four buckets matter. First, a task golden set: real support questions with expected answers, scored against a rubric. This measures the thing you actually trained for. Second, format adherence: does the output parse as valid JSON, keep the right tone, and stay within length limits? A correct answer in a broken format still breaks the product integration. Third, capability regression: run general benchmarks like MMLU on the base model and the fine-tuned model and compare. Aggressive training can make the model forget skills it used to have, which is called catastrophic forgetting. Fourth, safety and refusal: it should refuse disallowed requests, but not over-refuse harmless questions that merely look risky. Two options in the question are traps: GPU utilisation and final training loss describe the training run, not the model's behaviour, so they do not belong in an eval suite.

A post-fine-tune eval suite has to measure model behaviour on held-out data, never the training run itself. For a customer-support model, four orthogonal buckets are mandatory. First, a task golden set. Hand-curated support questions with expected answers, scored by rubric or an LLM judge. Second, format adherence: JSON parseability, brand tone, and length bounds. A correct answer in a broken format still fails the integration. Third, capability regression. Run a general suite like MMLU or GSM8K on base versus fine-tuned checkpoint to catch catastrophic forgetting. Fourth, safety in both directions. Measure under-refusal on disallowed asks and over-refusal on benign prompts that pattern-match dangerous ones. The two distractors test whether you confuse training signals with quality. GPU utilisation is an efficiency number for the run. Final training loss tracks the training distribution and barely predicts held-out behaviour. The unifying principle is that every eval set must be uncontaminated. If a test example leaked into training, the score is inflated and meaningless. You evaluate behaviour the model has never seen, across all four axes.