A product needs the model to answer from company specific knowledge. How you'd decide between RAG, fine-tuning, and long context: and why they're often combined rather than chosen exclusively.
Classify the gap first: fresh, large, citable knowledge → RAG; stable behavior or format → fine-tuning; small per-request docs → long-context. Mature systems combine all three.
Imagine teaching a new employee to handle your company's questions. If the answers live in a handbook that changes every week, you don't make them memorize it — you let them look it up each time, so they're never out of date. That's RAG. If they keep filling out forms wrong, you train them once so the right format becomes second nature. That's fine-tuning. And if a customer hands them one document for a single task, they just read it on the spot. That's long-context. A good system uses all three at once, because each fixes a different kind of problem.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Almost every team building a knowledge feature on top of an LLM hits this decision, and almost every interview that touches LLM system design asks some version of it. The framing — 'RAG, fine-tuning, or long-context?' — is a trap baked into the question, because it implies you must pick one. The strongest answers reject that framing and treat it as an allocation problem.
The reason the exclusive framing misleads is that the three tools don't compete for the same job. Fine-tuning changes how the model behaves. RAG changes which facts it can reach at query time. Long-context changes what it sees in this single prompt. Those are three different levers on three different gaps, and a real feature usually has more than one gap.
This deep dive builds a decision procedure that scales from the simple case to the combined one. We'll start with the gap classifier, then dig into the cost and failure mode that actually bites each option in production, then cover the handoff thresholds where one tool gives way to another, and finally show why mature systems run all three together.
Step one: decompose the feature into gaps
The first move a strong candidate makes is to refuse to answer 'which one' until the feature is broken into its gaps. A feature is rarely one need; it's a bundle.
There are two kinds of gap. A behavior gap is when the model knows enough but acts wrong — it ignores your output schema, drifts off your house tone, or handles a domain task clumsily. A knowledge gap is when the model lacks facts — your policies, current prices, a specific customer's records.
These map to different tools because they're different failures. You can't fix a behavior gap by handing the model more documents, and you can't fix a knowledge gap by changing how the model writes. So before choosing anything, list the gaps: 'it formats inconsistently' is behavior; 'it doesn't know our refund policy' is knowledge; 'the user pastes a log we must analyze' is per-request knowledge.
This decomposition is what makes the eventual 'combine them' answer principled rather than hand-wavy. You're not combining tools for the sake of sounding thorough; you're assigning one tool per gap because each gap has exactly one right kind of fix. The candidates who skip this step are the ones who get cornered into defending a single tool against a need it was never meant to address.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Dimension | RAG | Fine-tuning | Long-context |
|---|---|---|---|
| Gap it closes | Knowledge (large, fresh, citable) | Behavior / format / style | Small per-request knowledge |
| Freshness | Live — update the index | Stale until retrained | Fresh per request |
| Update cost | Reindex (cheap) | Training run (expensive) | None |
| Citations | Native (returns sources) | None | Possible if docs in prompt |
| Main cost / limit | Retrieval quality ceiling | Update cycle latency | Linear tokens; mid-prompt recall |
Real products, models, and research that use this idea.
- An enterprise assistant fine-tunes for a strict response schema, uses RAG over a docs index refreshed continuously, and drops user-pasted logs into long-context.
- Coding assistants combine RAG over a codebase with long-context for the open file and fine-tuning for diff/edit format conventions.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour RAG answers are wrong even though the documents contain the right facts. Where do you look first?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Framing it as a single exclusive choice — 'should we RAG or fine-tune?' — when the gaps are orthogonal and a real system layers all three on the parts each one fits.
60 second bullets to scan on the way to the call.
Why the question's exclusive framing is itself a trap
The two-step classifier: gap type, then volume and freshness
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.