What does RAG primarily help with in LLM-based applications?
RAG retrieves relevant docs and feeds them to the LLM as context, reducing hallucination on private or recent data.
Imagine asking a friend a question. They quickly look up notes, then answer based on those notes. That's RAG, the LLM is the friend, the notes are the retrieved documents.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Retrieval-Augmented Generation (RAG) is the standard architectural pattern for letting a frozen language model answer questions about information it was never trained on. The idea is simple, find the right documents, hand them to the model alongside the question, and let the model synthesize an answer grounded in those documents. The execution is where most of the engineering effort goes: how you chunk, how you embed, how you retrieve, how you rank, and how you measure.
RAG matters because the alternatives are worse for most real-world use cases. Re-pretraining or fine-tuning the model every time your data changes is expensive and slow. Stuffing all your documents into the context window is impossible at any meaningful corpus size. RAG lets the model stay frozen while the knowledge base evolves independently, a clean separation of concerns that pays off in production.
The three steps in detail
Retrieve, augment, generate. Retrieval starts with the user's query: you embed it into a vector (typically 1024-1536 dimensions) using the same model you used to embed your document chunks. You query a vector index for the top-k most similar chunks by cosine similarity (or hybrid score with BM25). Augment means assembling a prompt: the original query plus the retrieved chunks as context, framed by a system prompt instructing the model to cite sources and only use the provided context. Generate is the standard LLM completion, but now grounded in evidence you control.
The quality of each step compounds. Bad chunking poisons retrieval. Bad retrieval poisons augmentation. Bad prompt structure makes the model ignore the evidence and hallucinate from training data instead.
2–4 min · Everything important, quickly.
# Minimal RAG pipeline
docs = vector_store.similarity_search(query, k=5)
context = "\n\n".join(d.page_content for d in docs)
prompt = f"Use this context to answer:\n{context}\n\nQ: {query}"
answer = llm.invoke(prompt)| Aspect | RAG | Fine-tuning |
|---|---|---|
| Best for | Knowledge / facts | Style / format / skills |
| Update cost | Just re-index | Re-train (slow + costly) |
| Hallucination risk | Lower (grounded) | Same as base model |
| Setup complexity | Medium | High |
| Latency overhead | +retrieval (~100-500ms) | None |
Real products, models, and research that use this idea.
- Perplexity.ai retrieves web pages, reranks, then has Claude/GPT synthesize an answer with citations.
- GitHub Copilot Chat uses RAG over your open files and project context.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle a 10M-document corpus?
Sharded indexes + hierarchical retrieval (cluster routing then in-cluster ANN)
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Most candidates forget to mention reranking, which often beats retrieval choice for quality.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.