Fine-tuning on docs teaches the confident style of expert answers, not the facts themselves. The model gets surer of itself while staying ignorant, so it confabulates fluently.
Imagine a student who memorises how confident, polished medical answers sound but never studies the textbook. After a weekend of copying how doctors phrase things, they sound exactly like a doctor: calm, certain, full of jargon. But they still don't know the facts. So when you ask a real question, they answer in that same confident doctor voice and just make things up. Fine-tuning on a few thousand company documents does the same thing. It teaches the model the shape and tone of expert answers, not the underlying knowledge. The model becomes more sure of itself without becoming more correct. Confidence goes up, knowledge stays flat, and confident wrong answers are exactly what hallucination is.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
It feels obvious that fine-tuning a model on your company's documents should make it know those documents. The intuition is wrong often enough that it shows up as a senior interview question. In practice, teams fine-tune on internal wikis, runbooks, and support transcripts, then watch the model confidently invent API flags, policy clauses, and product facts that were never in the corpus. Hallucination goes up, not down.
The correct answer is A, and the reason is a mismatch between what supervised fine-tuning teaches and what teams assume it teaches. Fine-tuning is extremely good at teaching the SHAPE of an answer: the tone, the vocabulary, the confident structure of a domain expert. It is poor at teaching the FACTS, because the facts were never repeated at anything close to pretraining scale.
This deep dive walks the mechanism: the asymmetry between pretraining and fine-tuning, why the training objective inflates confidence, how catastrophic forgetting compounds the problem, why the three distractor mechanisms do not apply, and what the production fix actually looks like.
What supervised fine-tuning actually teaches
Supervised fine-tuning trains the model on instruction-response pairs. The gradient signal rewards reproducing the response given the instruction. Across a few thousand examples, what generalises most easily is the FORM of the responses: the register, the jargon, the confident cadence, the formatting conventions of the domain.
What does not generalise easily is the specific factual content. A given fact, say a parameter default or a policy threshold, appears once or twice in the training set. The model has no mechanism to durably memorise a fact from one or two exposures the way it memorised pretraining facts seen thousands of times across the web.
So the model emerges fluent in the domain's voice. It produces answers that LOOK like an expert wrote them. The style transferred; the knowledge mostly did not. The interview insight is recognising that style and knowledge are separable, and that fine-tuning preferentially transfers the easier one.
There is a deeper reason form transfers more readily than content. Style is a low-dimensional, highly repeated signal: every training example shares the same register, the same hedging patterns, the same way of opening and closing an answer. The model sees that pattern thousands of times even within a small dataset, so it locks in fast. Specific facts are the opposite: each one is a high-dimensional needle that appears in a single example, with no reinforcement from its neighbours. The gradient that would imprint that fact is tiny and easily washed out by the next batch.
This is why a fine-tune that looks like a triumph in a demo can be a liability in production. The demo questions overlap the training set, so the recalled style and the recalled facts coincide. Real users ask questions just outside that set, where the confident style is fully present but the matching fact never existed.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| What you want | Fine-tuning delivers | Better tool |
|---|---|---|
| Confident domain tone and format | Yes, efficiently from a few thousand examples | Fine-tuning |
| Memorising specific facts in the docs | Poorly; needs pretraining-scale repetition | RAG |
| Freshness when docs change | No; weights are stale at training time | RAG |
| Saying 'I don't know' on unknown queries | Only if you train abstention explicitly | Abstention fine-tune plus RAG |
Real products, models, and research that use this idea.
- Teams fine-tuning Llama 4 on internal wikis report it confidently inventing API parameters that sound right but never existed, fixed only by adding RAG over the live docs.
- OpenAI and Anthropic fine-tuning guides for GPT-5.5 and Claude Opus 4.7 steer customers toward retrieval for factual grounding and reserve fine-tuning for tone and format.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure whether a fine-tune raised hallucination rather than lowered it?
Build a held-out factual test set split into in distribution and out of distribution questions. Score accuracy and calibration before and after, and watch for confidence rising while accuracy stays flat or drops.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming fine-tuning on documents injects their facts into the model. It mostly injects the style of confident answers, which raises confabulation when the underlying knowledge never actually transferred.
60 second bullets to scan on the way to the call.
What supervised fine-tuning teaches well versus poorly
The asymmetry between pretraining scale and fine-tuning scale
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.