Your production RAG costs $1M/month. The CFO wants this cut in half with a max 1-point faithfulness regression. What highest leverage cost optimizations do you deploy, in priority order?
Production RAG cost is $1M/month. Quality budget: faithfulness can drop at most 1 point. Walk through the highest leverage cost optimizations in priority order. For each, name the expected cost reduction, the quality risk, and how you'd validate the change before rolling it out.
Free · 2 AI evals / day
Start with a cost breakdown. At $1M/month, a typical RAG stack splits roughly: LLM generation 50-65%, vector DB 10-20%, reranker 5-15%, embedding API 5-15%, infra 5-10%. The LLM is the biggest lever; optimize there first.
The six highest leverage moves are prompt caching, top-k reduction, vector quantization, generator routing to a smaller model on easy queries, semantic answer caching, and self-hosted Llama 4.
Ship them in this order: prompt caching first because it has zero quality risk and 2026 vendors all support it. Then top-k reduction, since fewer chunks in the prompt saves tokens and often improves answers. Then vector quantization, which cuts storage 4-8x with a recoverable recall hit. Then generator routing, which is the highest savings move but needs careful eval. Then semantic answer caching for repeated queries. Finally, self-hosted Llama 4 for sustained high volume traffic.
Gate every change on the golden eval set. Faithfulness must not drop more than 1 point.
**Profile first.** At $1M/month, the LLM generation eats 50-65% of the bill, vector DB 10-20%, reranker 5-15%, embedding 5-15%, infra 5-10%. Cost engineering follows the dominant line.
Six levers, ordered by (cost saved) / (validation effort).
1. Prompt caching. Anthropic, OpenAI, and Google all support it in 2026. System prompts and repeated chunks pay ~10% of normal cost. Savings: 20-40% on LLM cost. Quality risk: zero. Validate in shadow.
2. Top-k reduction. Drop from 15 chunks to 5-8 in the generator prompt. Fewer tokens means lower cost and often higher quality from less lost-in-the-middle dilution. Savings: 10-25% on LLM cost. Validate via faithfulness curve over k.
3. Vector quantization. Switch to product or scalar quantization. 4-8x storage cut with 1-3 points recall hit, recovered by reranker. Savings: 50-75% on vector DB cost. Validate via recall@k.
4. Generator routing. Route easy queries to Claude Haiku 4.7 or GPT-5.5-mini; hard queries to the frontier model. Savings: 30-50% on LLM cost. Quality risk is real; gate router thresholds on the eval.
5. Semantic answer caching. Cache full answers keyed by query embedding similarity. Hit rate 5-20%. Saves the whole pipeline on hits. Watch staleness.
6. Self-hosted Llama 4. At sustained load on H200/B200, 3-5x cheaper per million tokens than API. Validate with a 4-6 week parallel pilot.
Stacking 1, 2, 3, 4 typically hits 40-55% cost reduction within a quarter, comfortably inside the 1-point faithfulness budget if every change is gated on the golden eval.
Quick reference
TL;DR
Profile first; the LLM eats 50-65% of cost. Ship in order: prompt caching, top-k reduction, vector quantization, generator routing, semantic cache, self-hosted Llama. Gate each change on the eval set.
Think of your RAG bill like a household budget. Before you cut anything, you figure out where the money actually goes. In most homes the rent or mortgage is the biggest line item; in RAG, it is the smart-AI call that writes the answer. Once you know that, you attack the biggest bills first, but you start with the changes that cannot hurt quality: turn on the discount card the AI provider already gives you, send shorter shopping lists, and shrink your storage closet. Only then do you try the riskier moves, like using a cheaper helper for easy questions or running your own AI on your own machines. After each change, you check that the answers are still as honest as before. If they slip more than a tiny bit, you back the change out.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Spend is under $50K/month and stable The engineering and validation effort to ship six optimizations is not justified by the savings; focus on quality instead.
Traffic is bursty and unpredictable Self-hosted Llama 4 is uneconomic on bursty load; you pay for idle GPUs. Stay on API and route by tier.
Quality budget is zero regression on faithfulness Generator routing and quantization both carry small but nonzero risk; if the budget is exactly zero, only prompt caching and top-k reduction with reranker compensation are safe.
Corpus updates every few minutes Semantic answer caching becomes a staleness liability faster than it earns its cost savings; skip it until corpus update cadence slows.
Quick scan · 2–4 min
Everything important, quickly.
11 min: walk the profile, defend the ordering, give expected savings and validation for each of the six levers, and close with the 40-55% combined estimate.
Side-by-side
Optimization
Cost saved
Quality risk
Validation effort
Ship week
Prompt caching
20-40% of LLM
none
shadow A/B
1
Top-k reduction
10-25% of LLM
low (often improves)
faithfulness curve
2
Vector quantization
50-75% of vector DB
low (reranker recovers)
recall@k
3-4
Generator routing
30-50% of LLM
real, bounded
per-tier eval + A/B
month 2
Semantic answer cache
5-20% of pipeline
staleness
hit ratio + freshness
month 2
Self-hosted Llama 4
3-5x on volume tier
model quality gap
4-6 week pilot
month 3-4
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's prompt caching docs show 90% cost reduction on cached tokens, with a 5-minute default cache window suitable for RAG system prompts.
2OpenAI's prompt caching auto-caches repeated prefixes and is documented to cut cost 50% on cached portions.
3Pinecone serverless tiering moves cold vectors to cheaper storage automatically, often cutting vector DB cost 60-80% for large corpora.
4Anthropic Claude 4.7 contextual retrieval reduces the effective top-k needed because each chunk carries its own context, often dropping prompt size 30-40%.
5Self-hosted Llama 4 on B200 clusters with vLLM achieves 3-5x cost per token savings vs hosted APIs at sustained 100K+ TPS.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build the generator router classifier, and how do you tune its threshold against the 1-point faithfulness budget?
Sign in or go Pro to see the approach for this follow-up.
QHow do you decide when to switch from API to self-hosted Llama 4?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right TTL for semantic answer caching, and how do you invalidate on corpus updates?
Sign in or go Pro to see the approach for this follow-up.
QHow does prompt caching interact with conversational multi-turn RAG?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Jumping to fixes without profiling. If you do not know the cost breakdown, you optimize the wrong layer. The LLM dominates; embedding cost is rarely worth chasing first.