Why 'fine-tune on our company docs' usually fails: and what to do instead
A product manager asks you to fine-tune Llama-3 on the company's 50,000-page internal knowledge base so the model 'learns the company's knowledge'. Explain why this is the wrong approach and what to recommend instead.
Fine-tuning teaches behavior and style, not facts. To make a model know your docs, retrieve them with RAG; reserve a light fine-tune for voice and format.
Picture a brilliant new hire on day one. Fine-tuning is like sending them on an etiquette course: they come back speaking in your house style, formatting reports your way, refusing off-topic asks politely. The course does not paste 50,000 pages of policy into their memory. If you quiz them on a rule they only skimmed, they answer smoothly and wrongly, because they learned the voice, not the facts. RAG is the opposite move. You hand them the relevant pages right before each question, so they read and summarise from the real document. Want them to always sound on-brand and cite a source line? Send them to the short course. Want them to know what policy 7.3 says today? Give them the page. Smart teams do both.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: behavior vs knowledge framing + the four arguments against fine-tuning on docs + the concrete RAG pipeline + the light fine-tune slice + separate evaluation tracks.
| Concern | Fine-tune on docs | RAG over docs |
|---|---|---|
| Encodes | Voice, format, document structure | The actual facts, retrieved on demand |
| Fact recall | Unreliable, scale mismatch with pretraining | Grounded in the retrieved chunk |
| Freshness | Frozen at training time, needs re-train | Re-index in seconds to minutes |
| Hallucination | Confident confabulation on partial facts | Answer traceable to a cited source |
| General skill | Risk of catastrophic forgetting | Base model untouched |
Real products, models, and research that use this idea.
- Glean and other enterprise search copilots index company wikis with RAG over pgvector or Elasticsearch rather than fine-tuning a base model on the corpus.
- Perplexity grounds answers in live retrieval and supplies citations, then uses lightly tuned Claude Opus 4.7 and GPT-5.5 backends for synthesis style.
- OpenAI's fine-tuning API is marketed for tone, format, and JSON schema adherence on GPT-5.5-mini, while knowledge is expected to come from retrieval.
- Teams on Together.ai LoRA-fine-tune a Llama 4 variant for brand voice and a mandatory source line, then layer RAG with a BGE-reranker for facts.
- Notion and Confluence assistants retrieve from the workspace at query time so a policy edit is searchable instantly, with no model retrain required.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf fine-tuning cannot memorise facts at this scale, why does it sometimes appear to answer doc questions correctly?
QHow would you split the system so the fine-tune and the RAG layer are evaluated independently?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Believing fine-tuning injects facts into the model. It mostly teaches voice and format. On half-memorised facts the model confabulates fluently, which is worse than a base model that hedges.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.