A product manager asks you to fine-tune Llama-3 on the company's 50,000-page internal knowledge base so the model 'learns the company's knowledge'. Explain why this is the wrong approach and what to recommend instead.
Free · 2 AI evals / day
The request sounds reasonable but rests on a wrong mental model. **Fine-tuning** does not stuff facts into the model. It nudges the model's habits: tone, format, the way it reasons. It is great at HOW the model talks, weak at WHAT it knows.
Pretraining wrote facts into the weights using trillions of tokens. A 50,000-page corpus is a tiny fraction of that. Training on it teaches the corporate voice and document shape, but not reliable recall of every policy number or price.
Worse, on facts it only half-absorbed, the model answers smoothly and confidently while being wrong. A base model that says 'I am not sure' is safer than a fine-tuned one that confabulates in a believable tone.
The right tool for knowledge is RAG. Chunk the documents, embed them, store them in a vector database, and retrieve the most relevant pieces at query time. The model reads those pieces and answers, with citations you can trace.
Keep fine-tuning for what it is good at. A light fine-tune on a few hundred examples can lock in tone, format, and a 'Source:' line. The slogan: RAG for the knowledge, light fine-tune for the voice.
I would push back on the framing, not the ambition. **Fine-tuning** changes behavior, not knowledge. Training Llama-3 on 50,000 pages teaches the corporate voice and document structure. It does not reliably memorise the facts.
The reasons are structural. Pretraining used trillions of tokens to encode facts; this corpus is a few hundred million tokens, orders of magnitude smaller. On facts it only partly absorbed, the model confabulates fluently, which is worse than a base model that hedges. Docs also change weekly, but a fine-tune is frozen at training time. Aggressive training on a narrow corpus also risks catastrophic forgetting of general skill.
The right tool is RAG. Chunk the pages to a few hundred tokens, embed with a strong model, store in a vector database, and retrieve the top matches at query time. The model reads real text and answers with traceable citations. Updates are a re-index, not a retrain.
Fine-tuning still earns a small role. A light fine-tune on a few hundred examples locks in tone, format, and a source line. The recommendation: RAG for the facts, light fine-tune for the voice.
Quick reference
TL;DR
Fine-tuning teaches behavior and style, not facts. To make a model know your docs, retrieve them with RAG; reserve a light fine-tune for voice and format.
Picture a brilliant new hire on day one. Fine-tuning is like sending them on an etiquette course: they come back speaking in your house style, formatting reports your way, refusing off-topic asks politely. The course does not paste 50,000 pages of policy into their memory. If you quiz them on a rule they only skimmed, they answer smoothly and wrongly, because they learned the voice, not the facts. RAG is the opposite move. You hand them the relevant pages right before each question, so they read and summarise from the real document. Want them to always sound on-brand and cite a source line? Send them to the short course. Want them to know what policy 7.3 says today? Give them the page. Smart teams do both.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Internal documents that change every few days or hours A fine-tune freezes at training time, so only a live retrieval index keeps answers current.
Questions that demand a citation or audit trail to a source document Fine-tuned weights cannot point to which page produced a fact, but retrieved chunks can.
A general assistant employees rely on for broad reasoning Aggressive training on a narrow corpus risks catastrophic forgetting of general capability.
A corpus far larger than any context window can hold Retrieval acts as the compression step, surfacing only the relevant pages per query.
Quick scan · 2–4 min
Everything important, quickly.
4 min: behavior vs knowledge framing + the four arguments against fine-tuning on docs + the concrete RAG pipeline + the light fine-tune slice + separate evaluation tracks.
Side-by-side
Concern
Fine-tune on docs
RAG over docs
Encodes
Voice, format, document structure
The actual facts, retrieved on demand
Fact recall
Unreliable, scale mismatch with pretraining
Grounded in the retrieved chunk
Freshness
Frozen at training time, needs re-train
Re-index in seconds to minutes
Hallucination
Confident confabulation on partial facts
Answer traceable to a cited source
General skill
Risk of catastrophic forgetting
Base model untouched
Real-world examples
Real products, models, and research that use this idea.
1Glean and other enterprise search copilots index company wikis with RAG over pgvector or Elasticsearch rather than fine-tuning a base model on the corpus.
2Perplexity grounds answers in live retrieval and supplies citations, then uses lightly tuned Claude Opus 4.7 and GPT-5.5 backends for synthesis style.
3OpenAI's fine-tuning API is marketed for tone, format, and JSON schema adherence on GPT-5.5-mini, while knowledge is expected to come from retrieval.
4Teams on Together.ai LoRA-fine-tune a Llama 4 variant for brand voice and a mandatory source line, then layer RAG with a BGE-reranker for facts.
5Notion and Confluence assistants retrieve from the workspace at query time so a policy edit is searchable instantly, with no model retrain required.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf fine-tuning cannot memorise facts at this scale, why does it sometimes appear to answer doc questions correctly?
Sign in or go Pro to see the approach for this follow-up.
QHow would you split the system so the fine-tune and the RAG layer are evaluated independently?
Sign in or go Pro to see the approach for this follow-up.
QWhen would a light fine-tune actually improve RAG answer quality rather than just style?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing fine-tuning injects facts into the model. It mostly teaches voice and format. On half-memorised facts the model confabulates fluently, which is worse than a base model that hedges.