A post-fine-tune eval needs four behaviour axes: task quality on a golden set, format adherence, capability regression, and safety in both directions. Training metrics do not count.
Imagine you trained a new support agent. To grade them, you don't watch how much coffee they drank during training. That's effort, not skill. Instead you give a real exam. It has a set of customer questions with model answers (does the work get done?) and a check that their forms are filled out correctly (does the format hold?). It has a quiz on general knowledge to make sure training didn't make them forget basic things (did they forget anything?). Last come a few trick requests to see if they refuse the bad ones without being rude to the good ones (are they safe?). Training loss and how hard the computer worked describe the practice sessions, not the final exam. You only trust the exam, given on questions the agent never saw while training.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Of the six options, four describe the behaviour of the resulting model on data it has never seen. Two describe the training run. A post-fine-tune eval suite measures the former and ignores the latter, and confusing the two is a common evaluation mistake.
The training loop produces loud, satisfying numbers. The loss curve trends down, the GPUs run hot, throughput looks good, and it feels like progress. But none of those numbers answer the only question that matters after training: does the model now behave the way the product needs, on inputs it will actually see? A loss of 0.4 tells you the model fits the training distribution. It tells you almost nothing about whether the support bot answers a novel billing question correctly, emits parseable JSON, still knows basic arithmetic, or refuses a policy-violating ask without also stonewalling a benign one.
There is a deeper reason the four real buckets are non-negotiable: they are orthogonal. Each one catches a failure the others are blind to. A model can ace the golden set and still emit malformed JSON. It can hold its format perfectly and still have forgotten how to add two numbers. It can pass every capability benchmark and still refuse a customer asking how to cancel a subscription because the word 'terminate' looked dangerous. Because the failures are independent, you cannot collapse them into one score, and you cannot drop a bucket and hope another covers for it.
One principle ties all four together: every eval set must be uncontaminated and held out. The moment a test example leaks into training, the score measures memorisation rather than capability.
Bucket one: the task golden set
The golden set is the floor of any serious fine-tune eval. It is a hand-curated collection of representative support questions paired with expected answers and a scoring rubric. For a support model, that means realistic tickets across the real category mix: billing, account access, product how-to, escalation triggers, edge cases. The set should be stratified so the rare but important categories are not drowned out by the common ones. A model that nails password resets but botches every refund dispute can still post a deceptively high average.
Scoring runs two ways in practice. A human-labelled rubric gives the ground truth, and an LLM judge gives cheap scale. The rubric itself usually has several axes, such as factual correctness, completeness, and tone. Each is scored separately so a single weak dimension does not hide behind a strong one. You only trust the judge after correlating its scores against the human sample and checking for position and verbosity bias. The two should track before the judge replaces humans on the bulk of the set. Re-validate that agreement whenever the judge model changes underneath you.
The golden set measures the exact capability you fine-tuned for. If you skip it, you are optimising blind. You changed the weights to improve support answers and then never checked whether they improved on questions the model had not memorised. It is also the bucket that anchors the others, because a regression elsewhere only matters relative to whether the model still does its job.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Eval bucket | What it measures | Example metric |
|---|---|---|
| Task golden set | In-domain answer quality on held-out tickets | Rubric score, LLM-as-judge agreement |
| Format adherence | Output is parseable and on-brand | JSON parse rate, tone classifier, length bounds |
| Capability regression | General skills retained after tuning | MMLU, ARC, GSM8K delta versus base |
| Safety and refusal | Refuses bad asks, not benign ones | Refusal rate, XSTest over-refusal rate |
| Training loss / GPU (NOT eval) | Properties of the training run | Final loss, GPU utilisation, throughput |
Real products, models, and research that use this idea.
- OpenAI Evals is an open-source framework for writing held-out eval sets and graders, so a team can score a support fine-tune on task quality and format before it ships.
- XSTest (Röttger et al., NAACL 2024) gives benign prompts that look unsafe, so a team can report over-refusal next to the refusal rate on disallowed asks.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you build a golden set that resists contamination and stays representative over time?
Sample from real production traffic distinct from the training window, freeze and version it, hash examples against training data, and refresh quarterly as the ticket mix drifts.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting final training loss as the headline quality number. Loss tracks the training distribution and says almost nothing about behaviour on held-out tasks, format, or safety.
60 second bullets to scan on the way to the call.
The four held-out eval buckets and why each is orthogonal
Why training loss is a poor headline quality metric
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.