When you **fine-tune** a model, the dataset you trained on is NOT how you judge whether it worked. You need a separate eval suite that tests behaviour the model has never seen.
For a customer-support fine-tune, four buckets matter. First, a task golden set: real support questions with expected answers, scored against a rubric. This measures the thing you actually trained for.
Second, format adherence: does the output parse as valid JSON, keep the right tone, and stay within length limits? A correct answer in a broken format still breaks the product integration.
Third, capability regression: run general benchmarks like MMLU on the base model and the fine-tuned model and compare. Aggressive training can make the model forget skills it used to have, which is called catastrophic forgetting.
Fourth, safety and refusal: it should refuse disallowed requests, but not over-refuse harmless questions that merely look risky. Two options in the question are traps: GPU utilisation and final training loss describe the training run, not the model's behaviour, so they do not belong in an eval suite.
A post-fine-tune eval suite has to measure model behaviour on held-out data, never the training run itself. For a customer-support model, four orthogonal buckets are mandatory.
First, a task golden set. Hand-curated support questions with expected answers, scored by rubric or an LLM judge. Second, format adherence: JSON parseability, brand tone, and length bounds. A correct answer in a broken format still fails the integration. Third, capability regression. Run a general suite like MMLU or GSM8K on base versus fine-tuned checkpoint to catch catastrophic forgetting. Fourth, safety in both directions. Measure under-refusal on disallowed asks and over-refusal on benign prompts that pattern-match dangerous ones.
The two distractors test whether you confuse training signals with quality. GPU utilisation is an efficiency number for the run. Final training loss tracks the training distribution and barely predicts held-out behaviour.
The unifying principle is that every eval set must be uncontaminated. If a test example leaked into training, the score is inflated and meaningless. You evaluate behaviour the model has never seen, across all four axes.
Quick reference
TL;DR
A post-fine-tune eval needs four behaviour axes: task quality on a golden set, format adherence, capability regression, and safety in both directions. Training metrics do not count.
Imagine you trained a new support agent. To grade them, you don't watch how much coffee they drank during training, that's effort, not skill. Instead you give a real exam: a set of customer questions with model answers (does the work get done?), a check that their forms are filled out correctly (does the format hold?), a quiz on general knowledge to make sure training didn't make them forget basic things (did they forget anything?), and a few trick requests to see if they refuse the bad ones without being rude to the good ones (are they safe?). Training loss and how hard the computer worked describe the practice sessions, not the final exam. You only trust the exam, given on questions the agent never saw while training.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Using final training loss as the metric that decides whether to ship a checkpoint Loss tracks the training distribution and barely predicts held-out behaviour, so two equal-loss models can diverge sharply.
Putting GPU utilisation or throughput inside the model-quality eval suite These are efficiency numbers for the run and belong in a cost report, not a behaviour evaluation.
Scoring on examples that overlap with the training data Contamination inflates every number and hides the failures you most need to see before deploy.
Relying on a single LLM-judge score across all behaviour axes One blended number lets task quality, format, and safety mask each other so regressions go unseen.
Quick scan · 2–4 min
Everything important, quickly.
4 min: why eval is held-out behaviour not training signal + the four buckets (task, format, regression, safety) + contamination control + why loss and GPU are distractors.
Side-by-side
Eval bucket
What it measures
Example metric
Task golden set
In-domain answer quality on held-out tickets
Rubric score, LLM-as-judge agreement
Format adherence
Output is parseable and on-brand
JSON parse rate, tone classifier, length bounds
Capability regression
General skills retained after tuning
MMLU, ARC, GSM8K delta versus base
Safety and refusal
Refuses bad asks, not benign ones
Refusal rate, XSTest over-refusal rate
Training loss / GPU (NOT eval)
Properties of the training run
Final loss, GPU utilisation, throughput
Real-world examples
Real products, models, and research that use this idea.
1OpenAI Evals and the fine-tuning API workflow pair a held-out grading set with format checks before any GPT-4o-mini support tune ships.
2Anthropic publishes XSTest-style over-refusal probes so Claude Opus 4.7 safety tuning measures both refusal and over-refusal directions.
3LangSmith and Arize Phoenix expose golden-set scoring plus LLM-as-judge dashboards used to gate customer-support fine-tunes in production.
4Teams fine-tuning Llama 4 on Together.ai run MMLU and GSM8K on base versus tuned checkpoints to catch catastrophic forgetting before deploy.
5Hugging Face lighteval and the Open LLM Leaderboard suite are the standard regression harness for capability deltas after a domain fine-tune.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you build a golden set that resists contamination and stays representative over time?
Sign in or go Pro to see the approach for this follow-up.
QWhen can you trust an LLM-as-judge score instead of human grading on the support golden set?
Sign in or go Pro to see the approach for this follow-up.
QHow do you decide an acceptable capability-regression threshold rather than demanding zero drop?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reporting final training loss as the headline quality number. Loss tracks the training distribution and says almost nothing about behaviour on held-out tasks, format, or safety.