Which levers cut RAG serving cost without gutting answer quality?
Real RAG savings come from semantic caching, context compression, model routing, and index sizing. Dropping grounding directives or turning off eval saves pennies while removing safeguards: false economies.
Think of running a busy kitchen. The big costs are the chef's time and the ingredients you keep buying. You save real money in four ways. Keep a tray of popular dishes ready to reheat (caching). Trim each plate to only what the customer ordered (compression). Send simple orders to the junior cook and save the head chef for hard dishes (routing). Don't over-stock the pantry (index sizing). To save money, you should not stop tasting the food before it goes out (turning off evaluation). You also should not rip the safety labels off the ingredients (dropping the instructions that keep answers honest). Those cuts barely save anything and they let bad plates reach the customer.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Cost optimization is where RAG engineering separates the people who have run a system in production from the people who have only built a demo. A demo serves a handful of curated queries. Production serves a long, repetitive, messy tail at a token bill that scales with traffic. The instinct under cost pressure is to trim anything that looks like overhead. But some 'overhead' is a safeguard. Cutting it trades a real quality property for a saving so small it does not change the bill.
The right discipline is to profile first and cut where the money actually is, while protecting the signals that tell you quality is holding. The cost profile of a RAG request points to four real levers and two tempting false economies. Simple token arithmetic explains why each falls where it does.
Where the money actually goes
A single RAG request has a predictable cost breakdown. You embed the query (cheap, one small model call), search the vector index (cost grows with index size and dimensionality), and call the LLM to generate an answer from the retrieved context. That LLM call is almost always the dominant line item.
Within the LLM call, price is roughly linear in input tokens plus output tokens. In a RAG prompt the input is usually much larger than the output. You are stuffing several retrieved chunks, often thousands of tokens, to produce a few hundred tokens of answer. So input tokens are the biggest single driver of per-query cost.
Two other costs matter at scale. Redundant recomputation: if many queries are near-duplicates, you pay full price to regenerate answers you have effectively produced before. And the index itself: at a billion vectors, storage and per-query distance computation become a standing bill. Every real cost lever maps onto one of these: avoid the call, shrink its input, or shrink the index. Anything that does not touch those is not a serving-cost lever, no matter how it is presented.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Semantic caching layers like GPTCache return a stored answer when a new query is embedding-close to a past one, skipping the LLM call for the near-duplicate tail.
- LLMLingua compresses retrieved context by dropping low-information tokens, cutting prompt length while preserving the answer-bearing content.
What an interviewer would ask next. Try answering before peeking at the approach.
QYou apply all four levers at once. How do you tell which ones actually saved money and which quietly hurt quality?
Talk about isolating each lever behind a flag, measuring cost per query and faithfulness/answer-correctness before and after, and A/B testing on live traffic rather than trusting offline numbers alone.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Chasing 'savings' that remove safeguards. Cutting grounding directives saves a handful of tokens but raises hallucination. Turning off faithfulness eval removes the only signal that tells you quality is regressing.
60 second bullets to scan on the way to the call.
Identify the dominant cost components of a RAG serving request
Explain why semantic caching is often the single biggest saving
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.