**RAG** stands for **Retrieval-Augmented Generation**. The core idea is simple: instead of relying on what the LLM memorized during training, fetch relevant documents at query time and give them to the model as context.
Why this matters. An LLM's training data has a cutoff. It doesn't know about your private company docs, last week's news, or anything that happened after training finished. If you ask about those, it will either say it doesn't know or, worse, make something up. That's hallucination.
RAG fixes both failure modes by injecting real information into the prompt. The flow looks like: embed the user's query, search a vector database for relevant chunks, paste those chunks into the prompt as context, then let the LLM generate the answer using that context. The model is grounded in real text it can read, not just patterns it half remembers.
**RAG** retrieves relevant external documents at query time and feeds them to the LLM as context, grounding the answer in real, fresh, or private data.
The failure mode it addresses is specific: LLMs hallucinate on facts they didn't memorize well, and they have no way to know about anything past their training cutoff. RAG bypasses both. The model doesn't need to have memorized your company's wiki; the retrieval step finds the right wiki page and hands it over.
The canonical pipeline is five steps: embed the query, vector-search for top-k chunks, optionally rerank, stuff the surviving chunks into the prompt, then generate.
Where RAG wins: private knowledge (internal docs, user history, proprietary data), recent knowledge (news, ticket queues, evolving documentation), and **citable answers**. Where it loses: queries that don't reference external facts, ultra low latency chat, or knowledge so small it fits in the context window directly. The classic comparison is RAG vs fine-tuning, RAG handles fresh facts, fine-tuning handles style and reasoning patterns.
Quick reference
TL;DR
RAG retrieves relevant docs at query time and stuffs them into the prompt, grounding the LLM's answer in real, fresh, or private data instead of hoping it memorized the fact.
Imagine asking a smart friend a question about a niche topic they don't know much about. Without help, they'll guess (often confidently). Now give the same friend a few relevant pages from a textbook before they answer. They read the pages, then respond using both their general knowledge and the pages in front of them. That's RAG, short for Retrieval-Augmented Generation. The friend is the LLM. The pages are documents fetched from a vector database. The fetching happens automatically at query time, so the model always has fresh, specific information for whatever the user just asked.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Queries that don't reference external facts Pure reasoning queries pull in noise from retrieval; the LLM does better with no extra context to filter through.
Tight budget interactive chat under 100ms p99 Embedding, search, rerank, and prompt assembly each cost tens of milliseconds; budget falls apart fast.
Corpus small enough to fit in the prompt If the entire knowledge base is 30k tokens, just paste it; the retrieval machinery is overhead without payoff.
Strict consistency knowledge that changes by the second Vector index lag (minutes to hours) means RAG can serve stale facts; consider live call architecture or write through indexing.
Quick scan · 2–4 min
Everything important, quickly.
3 min: 5-step pipeline + RAG vs fine-tune + when each wins + production gotchas + one eval framework.
Side-by-side
Concern
RAG
Fine-tuning
Fresh / changing data
Just reindex, no retraining
Have to retrain (slow, expensive)
Style or format learning
Weak, prompt level only
Strong, baked into weights
Citable answers
Yes, chunks have provenance
No, weights have no source
Cost per query
Higher, retrieval adds tokens
Lower, no retrieval step
Best for
Private or recent facts
Behavior, voice, domain reasoning
Real-world examples
Real products, models, and research that use this idea.
1Perplexity retrieves web pages, reranks, and asks Claude or GPT to synthesize an answer with citations.
2Notion AI runs RAG over your workspace docs to answer questions grounded in your own content.
3GitHub Copilot Chat retrieves chunks from your open files and project to ground code completion responses.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you eval whether RAG is actually helping vs hurting?
Sign in or go Pro to see the approach for this follow-up.
QWhat's your strategy for the case where retrieval surfaces low-confidence chunks?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you combine RAG with fine-tuning instead of picking one?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing RAG and fine-tuning solve the same problem. RAG injects fresh facts; fine-tuning teaches style, format, or domain reasoning.