Predict the prompt token count before and after context compression
A RAG prompt packs k = 8 retrieved chunks at T = 500 tokens each, plus a fixed overhead of 300 tokens for the system prompt, question, and formatting. A contextual compressor keeps a fraction f = 0.4 of each chunk's tokens (overhead is unaffected). Give the total prompt tokens before compression and after compression.
Before = 4300 tokens (8×500 + 300). After = 1900 tokens: compression scales only the 4000 chunk tokens by 0.4 to 1600, while the 300 fixed overhead doesn't shrink.
Imagine packing a suitcase for a trip. Most of the space is clothes, but there's also a fixed block — your toiletry bag and shoes — that you can't squish. You have eight outfits, each taking up a certain amount of room, plus that fixed block. Now a clever packing service vacuum-seals only the clothes down to 40 percent of their bulk; it won't touch the toiletry bag or shoes. So the clothes shrink a lot, but the fixed block rides along unchanged. When you add it all back up, the suitcase is much smaller than before — but not 40 percent of the original, because that un-squishable block is still in there taking its full space. The fixed part is why the savings never reach the full headline number.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This looks like a one-line arithmetic question, and the number is easy — before = 4300, after = 1900. The reason it earns an interview slot is the distractor baked into the setup: there are two cost terms, and the compression fraction applies to only one of them.
Candidates who pattern-match "multiply by 0.4" onto the whole prompt get 1720 and miss the point entirely. The real lesson is how a fixed overhead dilutes an advertised compression ratio, which is exactly the kind of back of envelope reasoning that decides whether a token-optimization is worth shipping. We'll do the arithmetic, then generalize it into something you can reuse on any prompt-cost estimate.
Decompose the prompt before touching the numbers
Every RAG prompt has two kinds of tokens, and conflating them is the source of most cost-estimate errors. There's the variable retrieval payload — the chunks, which scale with how many you pack (k) and how big each is (T). And there's the fixed scaffolding — the system prompt, the user's question, formatting and delimiters — which is roughly constant regardless of retrieval.
In this scenario the chunk payload is k × T = 8 × 500 = 4000 tokens, and the fixed overhead is 300 tokens. Total before compression: 4000 + 300 = 4300 tokens.
The discipline of writing it as two terms, not one lump, is what protects you from the trap in the next step. A contextual compressor operates on retrieved content — it prunes or summarizes the chunks. It has no reason and usually no ability to rewrite your system prompt or drop the user's question. So the moment you split the prompt into compressible and fixed terms, it's obvious which term the fraction multiplies and which one is invariant. Skip the decomposition and you'll reflexively scale the 4300 total and land on a wrong number.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LLMLingua-style contextual compressors prune retrieved chunks before the prompt, scaling only the chunk budget while the system prompt stays fixed.
- A RAG product packing 8 chunks of 500 tokens against a 300-token system scaffold sees a 4300 → 1900 token drop per call after 0.4 compression.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the headline versus realized gap change if the overhead were 1500 tokens instead of 300?
Recompute after = 1600 + 1500 = 3100 against a before of 5500; the realized reduction shrinks because the fixed term is now a larger share of the prompt.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Multiplying the whole 4300-token prompt by 0.4 instead of only the 4000 chunk tokens. That gives 1720, ignoring that the 300-token overhead doesn't compress — the right answer is 1900.
60 second bullets to scan on the way to the call.
Decomposing a prompt into a variable chunk term and a fixed overhead
Why the compression fraction applies only to the chunk tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.