Explain when fine-tuning is the better choice over RAG, and vice versa.
Fine-tune to change behavior, style, or domain reasoning. Use RAG when you need fresh or private facts. Most production systems use both.
Imagine teaching a smart assistant. Fine-tuning is like sending it to a training course that rewires its habits. After the course, it talks, formats, and reasons differently forever. RAG is more like handing it a folder of notes right before each question. The assistant itself is unchanged, but it can read fresh facts from the folder. If you want your assistant to always answer in a polite legal tone, you train it. If you want it to know what your company shipped yesterday, you give it the folder. Most real systems do both: train the tone and hand over the folder.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Fine-tuning versus RAG is rarely a real either-or. It feels like one because the two techniques are introduced in parallel chapters of every LLM course. In production they solve different problems, and most serious stacks ship both.
Teams often burn weeks building one when the problem needed the other. A team will fine-tune a model to know their internal docs, then watch the model hallucinate when the docs change a week later. Another team will pile retrieved context onto every query to enforce a JSON schema. A 30-minute LoRA fine-tune would have solved it permanently. Knowing which side of the line a problem sits on is the key judgment call.
The ambiguity goes away once you treat them as edits to different parts of the system instead of competing 'approaches'. Fine-tuning edits the parameters. RAG edits the prompt. Those are two different operations on two different artifacts, and they produce two different categories of behavior change. Anything that needs gradient signal goes to fine-tuning; anything that needs fresh text goes to RAG.
What fine-tuning changes
Fine-tuning updates the model's parameters using gradient descent on labelled examples. Whether you use full fine-tuning, LoRA, or QLoRA, the artifact at the end is a modified set of weights. That artifact captures BEHAVIOR, the model's output distribution conditioned on input, generalised across many examples.
The behaviors that fine-tuning encodes well: tone and persona, structured output adherence like JSON or XML, and refusal policies. It also encodes domain reasoning patterns that need new internal abstractions (medical triage flows, legal citation formatting), tool-use and function-calling syntax, and translation between specialised registers.
What fine-tuning CANNOT do well: hold a specific fact. Memorising the price of widget X with fine-tuning is wasteful, because you have to retrain when the price changes. It is also unreliable, because the model may parrot a similar widget Y's price. Facts belong in retrieval; behaviors belong in weights. The clearest test: if updating the answer needs retraining, you put the wrong thing in weights.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Concern | Fine-tuning | RAG |
|---|---|---|
| Best for | Behavior, style, format, domain reasoning | Fresh or private facts, citations, large corpora |
| Update cadence | Days (data, train, eval, deploy) | Seconds to minutes (reindex) |
| Latency | Lower (no retrieval hop) | Higher (retrieval adds 100-300ms) |
| Per-query cost | Lower (smaller prompt) | Higher (retrieved context inflates tokens) |
| Operational complexity | Training pipeline, eval suite | Index, retriever, reranker, freshness |
Real products, models, and research that use this idea.
- Perplexity's Sonar model (2025) is built on Llama 3.3 70B and further trained for answer factuality and readability. It answers over live web results retrieved for each query.
- GitHub Copilot Chat retrieves from open files and project context and adds them to the prompt, so the model sees current code without retraining.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can't you use RAG to teach a model to output strict JSON?
Retrieval changes the prompt, not the decoding distribution. Schema adherence is a behavioral property that needs either gradient updates or constrained decoding, not more context.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating RAG and fine-tuning as competing solutions to the same problem. They solve different layers: behavior versus knowledge. The strongest production stacks combine both.
60 second bullets to scan on the way to the call.
Behavior versus knowledge framing
When fine-tuning is the only viable option
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.