Why 'fine-tune on our company docs' usually fails: and what to do instead
A product manager asks you to fine-tune Llama-3 on the company's 50,000 page internal knowledge base so the model 'learns the company's knowledge'. Explain why this is the wrong approach and what to recommend instead.
Fine-tuning teaches behavior and style, not facts. To make a model know your docs, retrieve them with RAG; reserve a light fine-tune for voice and format.
Picture a brilliant new hire on day one. Fine-tuning is like sending them on an etiquette course: they come back speaking in your house style, formatting reports your way, refusing off-topic asks politely. The course does not paste 50,000 pages of policy into their memory. If you quiz them on a rule they only skimmed, they answer smoothly and wrongly, because they learned the voice, not the facts. RAG is the opposite move. You hand them the relevant pages right before each question, so they read and summarise from the real document. Want them to always sound on-brand and cite a source line? Send them to the short course. Want them to know what policy 7.3 says today? Give them the page. Smart teams do both.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The instinct behind 'fine-tune on our docs so the model learns our knowledge' is natural, and it is wrong in a specific, instructive way. The request conflates two operations that live in different parts of the system. Fine-tuning edits the model's parameters, which encode BEHAVIOR. Retrieval edits the prompt at query time, which is how you supply KNOWLEDGE. The product manager wants knowledge, so the parameter edit is the wrong lever.
The reason this matters is that teams routinely spend weeks fine-tuning on a corpus, ship a model that sounds perfectly on-brand, and then watch it answer factual questions fluently and wrongly. The fluency is the trap. The model learned the corporate voice and the shape of the documents, which is exactly what fine-tuning is good at, and people mistake that polish for knowledge.
The test that cuts through the confusion is simple. Ask what happens when the answer changes. If updating an answer requires retraining the model, you put the wrong thing in the weights. Prices, policies, account records, and yesterday's announcement all change without warning, so they belong in a layer you can edit in seconds. Tone, format, and refusal style are stable, so they belong in the weights.
This deep dive walks the four structural arguments against fine-tuning for knowledge, lays out the RAG pipeline that actually solves the problem, and then carves out the genuine slice where a light fine-tune still earns its keep. The goal is the senior answer: RAG for the facts, light fine-tune for the voice.
Why fine-tuning cannot carry the knowledge
Start with scale. Pretraining wove facts into the weights using trillions of tokens, repeated and reinforced across the corpus. A 50,000-page knowledge base is perhaps a few hundred million tokens, three to four orders of magnitude smaller. Supervised fine-tuning at that scale does not reliably memorise the underlying facts. It teaches which tokens to use in which style.
Next, the failure mode. On a specific factual question the model only partly absorbed, it does not say 'I do not know'. It produces a fluent, confident, corporate-sounding answer that is subtly wrong. This confident confabulation is worse than a base model that visibly hedges, because it is harder to catch in review.
Then freshness. Internal docs change constantly, and a fine-tuned model is frozen at training time. A single policy edit would require a fresh fine-tune to propagate, which is operationally absurd at any real update cadence.
Finally, the side effect. Aggressive fine-tuning on a narrow corporate corpus can degrade general capability, the catastrophic forgetting risk. You can make the model worse at the broad reasoning tasks employees actually rely on it for.
There is a deeper mechanism worth naming. A fact is reliably retrievable from weights only when it appears many times across the training distribution, so the gradient signal reinforces it repeatedly. A policy number that shows up once in 50,000 pages gets a vanishingly small share of the training signal. The model learns the surrounding phrasing far more strongly than the specific value. That is why fine-tuned models nail the cadence of a policy sentence yet drop or swap the actual number inside it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Concern | Fine-tune on docs | RAG over docs |
|---|---|---|
| Encodes | Voice, format, document structure | The actual facts, retrieved on demand |
| Fact recall | Unreliable, scale mismatch with pretraining | Grounded in the retrieved chunk |
| Freshness | Frozen at training time, needs re-train | Re-index in seconds to minutes |
| Hallucination | Confident confabulation on partial facts | Answer traceable to a cited source |
| General skill | Risk of catastrophic forgetting | Base model untouched |
Real products, models, and research that use this idea.
- Glean and other enterprise search copilots index company wikis with RAG over pgvector or Elasticsearch rather than fine-tuning a base model on the corpus.
- Perplexity grounds answers in live retrieval and supplies citations, then uses lightly tuned Claude Opus 4.7 and GPT-5.5 backends for synthesis style.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf fine-tuning cannot memorise facts at this scale, why does it sometimes appear to answer doc questions correctly?
Separate genuine recall from style mimicry. The model often reproduces phrasings and high-frequency facts seen many times, while failing on long-tail specifics. Probe with rare facts to expose the gap.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing fine-tuning injects facts into the model. It mostly teaches voice and format. On half-memorised facts the model confabulates fluently, which is worse than a base model that hedges.
60 second bullets to scan on the way to the call.
Why fine-tuning encodes behavior rather than facts
The token scale mismatch versus pretraining
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.