Which levers cut RAG serving cost without gutting answer quality?

Multi-select·Medium·4.0 · 0·~1 min·Asked atHugging Face Mistral AI

Attempt it

To cut the cost of a RAG system, first figure out where the money actually goes. The big expenses are generation tokens (the LLM call), repeated work on queries you have effectively answered before, and the size and per-query cost of the vector index. Good cost levers shrink those without changing what makes an answer correct. Four options do exactly that. Semantic caching reuses a prior answer for a repeated or reworded query, skipping the LLM call entirely. Context compression strips each retrieved chunk down to the sentences that actually matter, cutting prompt tokens. Model routing sends easy questions to a small cheap model and saves the expensive large model for hard ones. Tightening top-k and quantizing vectors shrinks both per-query work and storage. The two wrong options are false economies. Dropping grounding directives from the prompt saves only a few tokens but makes the model more likely to hallucinate. Turning off the faithfulness evaluation saves a little offline compute but removes the signal that warns you when quality drops. Both trade away a safeguard for almost no saving.

**The discipline:** spend a token budget where it buys quality and cut it where it doesn't. Profile the request first. The dominant costs in RAG serving are LLM generation tokens, redundant recomputation, and the vector index footprint. The four genuine levers map onto those costs. Semantic caching keys on embedding similarity so paraphrased repeats hit a stored answer and skip retrieval plus generation. It is usually the biggest single win because a long tail of queries are near-duplicates. Context compression keeps only the query-relevant sentences from each chunk, cutting prompt tokens and, as a bonus, reducing lost-in-the-middle distractor noise. Model routing classifies query difficulty and sends the easy majority to a cheaper generator while reserving the large model for genuinely hard queries. Tightening top-k and quantizing vectors trims per-query distance computation and storage. The two distractors are traps. Grounding directives are a handful of instruction tokens; removing them saves almost nothing while raising hallucination rate — you trade a real quality property for a rounding-error of cost. Turning off the offline faithfulness evaluation saves a little batch compute but is not a serving-cost lever at all, and it blinds you to regressions. The interview signal is distinguishing levers that attack the expensive parts of the pipeline from cuts that merely remove safeguards.

TL;DR

Real RAG savings come from semantic caching, context compression, model routing, and index sizing. Dropping grounding directives or turning off eval saves pennies while removing safeguards — false economies.

Memory aid

Easy to grasp

Think of running a busy kitchen. The big costs are the chef's time and the ingredients you keep buying. You save real money by keeping a tray of popular dishes ready to reheat (caching), trimming each plate to only what the customer ordered (compression), sending simple orders to the junior cook and saving the head chef for hard dishes (routing), and not over-stocking the pantry (index sizing). What you should not do to save money is stop tasting the food before it goes out (turning off evaluation) or rip the safety labels off the ingredients (dropping the instructions that keep answers honest). Those cuts barely save anything and they let bad plates reach the customer.

Key concepts

Retrieval-Augmented Generation Inference Optimization LLM System Design

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Cost optimization is where RAG engineering separates the people who have run a system in production from the people who have only built a demo. A demo serves a handful of curated queries; production serves a long, repetitive, messy tail at a token bill that scales with traffic. The instinct under cost pressure is to trim anything that looks like overhead — but some 'overhead' is a safeguard, and cutting it trades a real quality property for a saving so small it does not move the bill.

The right discipline is to profile first and cut where the money actually is, while protecting the signals that tell you quality is holding. This walkthrough builds the cost profile of a RAG request, then works through the four genuine levers and the two seductive false economies, with the numbers that explain why each falls where it does.

Where the money actually goes

A single RAG request has a predictable cost breakdown. You embed the query (cheap, one small model call), search the vector index (cost grows with index size and dimensionality), and call the LLM to generate an answer from the retrieved context. That LLM call is almost always the dominant line item.

Within the LLM call, price is roughly linear in input tokens plus output tokens, and in a RAG prompt the input usually dwarfs the output: you are stuffing several retrieved chunks, often thousands of tokens, to produce a few hundred tokens of answer. So input tokens are the biggest single driver of per-query cost.

Two other costs matter at scale. Redundant recomputation: if many queries are near-duplicates, you pay full price to regenerate answers you have effectively produced before. And the index itself: at a billion vectors, storage and per-query distance computation become a standing bill. Every real cost lever maps onto one of these — avoid the call, shrink its input, or shrink the index. Anything that does not touch those is not a serving-cost lever, no matter how it is dressed up.

Caching, compression, routing, sizing

Why the two distractors are false economies

Optimizing safely with an eval harness

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Semantic caching layers like GPTCache return a stored answer when a new query is embedding-close to a past one, skipping the LLM call for the near-duplicate tail.
LLMLingua compresses retrieved context by dropping low-information tokens, cutting prompt length while preserving the answer-bearing content.

What an interviewer would ask next. Try answering before peeking at the approach.

QYou apply all four levers at once — how do you tell which ones actually saved money and which quietly hurt quality?

Talk about isolating each lever behind a flag, measuring cost per query and faithfulness/answer-correctness before and after, and A/B testing on live traffic rather than trusting offline numbers alone.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Chasing 'savings' that remove safeguards — cutting grounding directives saves a handful of tokens but raises hallucination, and turning off faithfulness eval removes the only signal that tells you quality is regressing.

60 second bullets to scan on the way to the call.

Identify the dominant cost components of a RAG serving request
Explain why semantic caching is often the single biggest saving

Primary sources. Browse if you want the original framing.