Long context vs RAG
Show the model everything, or choose what it should see
Long context lets the model see everything; RAG selects what it should see. Long context is convenient but not free: cost scales, and quality falls off past the middle of the window.
Long context
Glossary →Send a very large amount of text (documents, transcripts, code) directly to a model with a big context window. No retrieval, no vector database, no chunking pipeline; the model itself has to attend to everything you give it.
Best for: One-shot analysis of a single large document.
RAG
Glossary →Retrieval-augmented generation. Store documents in a vector database (and often a keyword index), retrieve a shortlist per query, and hand only the relevant chunks to the model. The model's context stays small; the pipeline chooses well.
Best for: Many documents, high freshness, cost per query matters.
At a glance
| Dimension | Long context | RAG |
|---|---|---|
| Cost per query | Scales with input length | Roughly flat (retrieval plus small context) |
| Freshness | Re-send everything each time | Re-index only what changed |
| Latency | Grows with input length | Retrieval plus small generation |
| Quality on long docs | Falls off past the middle | Stable when retrieval is good |
| Citations | Hard (fuzzy origin) | Easy (per-chunk provenance) |
| Best for | Ad-hoc analysis of one doc | Multi-doc, high-freshness, cost-sensitive |
Key differences
- 1Long context pays per token every query; RAG retrieves cheaply and only sends the shortlist
- 2Long context degrades in the middle of the window (lost-in-the-middle); RAG keeps grounding tight
- 3Long context can't be updated without resending everything; RAG updates by re-indexing
- 4RAG lets you cite exactly which chunk grounded a claim; long context blurs where the answer came from
- 5Long context latency grows with input size; RAG stays roughly flat
In the interview
- Claiming a 1M-token window makes RAG obsolete
- Missing lost-in-the-middle degradation
- Ignoring that long context is billed per token, per query
- Forgetting that RAG makes provenance easy and long context does not
How to choose
One doc, one shot → long context. Many docs, ongoing, cost-sensitive → RAG.
Common misconceptions
Myth: A million-token window kills RAG.
Reality: Quality still degrades past the middle, cost scales linearly per query, and freshness across many documents is far cheaper to keep via a vector index.
Myth: RAG is only for old, small models.
Reality: Modern production stacks still lean on RAG because it stays cheap and citable regardless of the context ceiling.
Memory aid
Long context is putting the whole library in your head; RAG is knowing which shelf to walk to.
Can you combine them?
Yes. The mature pattern is RAG with a moderately long context: retrieve the top chunks, then let the model attend to a healthy fraction of the window. Long context earns its keep as the ceiling, not the routine.
Related topics
Related comparisons
Bi-encoder vs Cross-encoder
Two ways embeddings power retrieval and reranking
Chunking strategies: fixed vs semantic vs recursive
Three ways to slice documents before they hit the vector store
Hallucination Mitigation vs Context Window Management
Two critical challenges in production LLM systems
RAG vs Fine-tuning
Two approaches to giving LLMs domain specific knowledge
Reranker vs Retriever
The two stages of production RAG, and why you need both
Semantic Search vs Keyword Search
Dense vector similarity vs sparse term matching for retrieval