Cutting RAG prompt tokens without dropping whole retrieved chunks: how does context compression work?
Describe contextual (context) compression in a RAG pipeline. How does it reduce prompt size without discarding entire chunks, and what does it buy you beyond raw token cost?
Context compression trims each retrieved chunk down to its query-relevant sentences or tokens before the prompt — cutting cost and latency while raising the signal to noise of the context the model reads.
Imagine you ask a friend a question and they grab three thick library books that mention your topic. Instead of reading all three cover to cover, they highlight only the few lines that actually answer you and read just those out loud. That is context compression in RAG. The search step pulls back big passages, but most of each passage is unrelated padding. Before handing them to the model, a smaller helper keeps the highlighted lines and throws away the rest. You pay for fewer words, you get the answer faster, and — surprisingly — the model often answers better, because it no longer has to wade through pages of filler to find the one fact that matters.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every RAG team eventually hits the same wall. Retrieval is tuned to recall, so you set top-k generously and pull back ten or twenty chunks to be safe. Now the prompt is huge, the bill climbs with every query, and — more quietly — answer quality starts to wobble even though the right chunk is definitely in there somewhere.
Context compression is the lever that addresses both problems at once. The interview question is rarely "what is it" — it is "why does trimming the context sometimes make the model more accurate, not just cheaper?" That second-order effect is the whole reason the technique is interesting, and it is where strong candidates separate from ones who only memorized the definition.
This deep dive walks through where compression sits, the two mechanism families and their trade-offs, the quality argument grounded in how attention behaves, and the boundary where long-context models make the whole thing optional.
The chunk-granularity mismatch that creates the problem
Start with why there is anything to compress. A retriever scores and returns chunks, because chunks are the unit you embedded and indexed. A chunk is sized for retrieval — say 300 to 500 tokens with some overlap — to be self-contained enough to embed meaningfully.
But the question a user asks is almost never about the whole chunk. It is about one fact, one definition, one number that happens to live inside a paragraph surrounded by setup, caveats, and unrelated neighboring sentences. The retriever cannot help this; it has no way to return half a chunk.
So the prompt you assemble carries a lot of text that was relevant to the chunk's author but is filler relative to this query. Multiply that across ten retrieved chunks and most of your context window is padding.
Context compression closes the gap. It re-examines each retrieved chunk against the actual query and keeps only the spans that earn their place. The granularity of relevance — sentence or token — finally matches the granularity of what reaches the model, instead of being stuck at whatever size you chose for indexing.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LLMLingua and LongLLMLingua compress prompts at the token level using a small model's perplexity, commonly hitting 2x to 5x reduction with little accuracy loss.
- LangChain ships a ContextualCompressionRetriever that wraps a base retriever with an LLM or embedding-based filter to trim retrieved chunks.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you tune the compression ratio without silently dropping answer-bearing evidence?
Sweep the ratio against a labelled eval set, tracking faithfulness and answer relevance plus a context-recall metric; pick the point where recall holds and stop before it falls.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating compression as fixed truncation (keep the first N tokens) instead of a query-conditioned step that decides what to keep based on the actual question.
60 second bullets to scan on the way to the call.
Where compression sits in the pipeline and why it is post-retrieval
The difference between extractive filtering and token-level compression
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.