Your production RAG costs $1M/month. The CFO wants this cut in half with a max 1-point faithfulness regression. What highest leverage cost optimizations do you deploy, in priority order?

Short answer·Hard·4.0 · 0·~3 min·Relevant atAnthropic Databricks OpenAI

Attempt it

Production RAG cost is $1M/month. Quality budget: faithfulness can drop at most 1 point. Walk through the highest leverage cost optimizations in priority order. For each, name the expected cost reduction, the quality risk, and how you'd validate the change before rolling it out.

Free · 2 AI evals / day

Start with a cost breakdown. At $1M/month, a typical RAG stack splits roughly: LLM generation 50-65%, vector DB 10-20%, reranker 5-15%, embedding API 5-15%, infra 5-10%. The LLM is the biggest lever; optimize there first. The six highest leverage moves are prompt caching, top-k reduction, vector quantization, generator routing to a smaller model on easy queries, semantic answer caching, and self-hosted Llama 4. Ship them in this order: prompt caching first because it has zero quality risk and 2026 vendors all support it. Then top-k reduction, since fewer chunks in the prompt saves tokens and often improves answers. Then vector quantization, which cuts storage 4-8x with a recoverable recall hit. Then generator routing, which is the highest savings move but needs careful eval. Then semantic answer caching for repeated queries. Finally, self-hosted Llama 4 for sustained high volume traffic. Gate every change on the golden eval set. Faithfulness must not drop more than 1 point.

**Profile first.** At $1M/month, the LLM generation eats 50-65% of the bill, vector DB 10-20%, reranker 5-15%, embedding 5-15%, infra 5-10%. Cost engineering follows the dominant line. Six levers, ordered by (cost saved) / (validation effort). 1. Prompt caching. Anthropic, OpenAI, and Google all support it in 2026. System prompts and repeated chunks pay ~10% of normal cost. Savings: 20-40% on LLM cost. Quality risk: zero. Validate in shadow. 2. Top-k reduction. Drop from 15 chunks to 5-8 in the generator prompt. Fewer tokens means lower cost and often higher quality from less lost-in-the-middle dilution. Savings: 10-25% on LLM cost. Validate via faithfulness curve over k. 3. Vector quantization. Switch to product or scalar quantization. 4-8x storage cut with 1-3 points recall hit, recovered by reranker. Savings: 50-75% on vector DB cost. Validate via recall@k. 4. Generator routing. Route easy queries to Claude Haiku 4.7 or GPT-5.5-mini; hard queries to the frontier model. Savings: 30-50% on LLM cost. Quality risk is real; gate router thresholds on the eval. 5. Semantic answer caching. Cache full answers keyed by query embedding similarity. Hit rate 5-20%. Saves the whole pipeline on hits. Watch staleness. 6. Self-hosted Llama 4. At sustained load on H200/B200, 3-5x cheaper per million tokens than API. Validate with a 4-6 week parallel pilot. Stacking 1, 2, 3, 4 typically hits 40-55% cost reduction within a quarter, comfortably inside the 1-point faithfulness budget if every change is gated on the golden eval.

Quick reference

TL;DR

Profile first; the LLM eats 50-65% of cost. Ship in order: prompt caching, top-k reduction, vector quantization, generator routing, semantic cache, self-hosted Llama. Gate each change on the eval set.

Memory aid

Easy to grasp

Think of your RAG bill like a household budget. Before you cut anything, you figure out where the money actually goes. In most homes the rent or mortgage is the biggest line item; in RAG, it is the smart-AI call that writes the answer. Once you know that, you attack the biggest bills first, but you start with the changes that cannot hurt quality: turn on the discount card the AI provider already gives you, send shorter shopping lists, and shrink your storage closet. Only then do you try the riskier moves, like using a cheaper helper for easy questions or running your own AI on your own machines. After each change, you check that the answers are still as honest as before. If they slip more than a tiny bit, you back the change out.

Key concepts

RAG system design Vector databases Embeddings

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

11 min: walk the profile, defend the ordering, give expected savings and validation for each of the six levers, and close with the 40-55% combined estimate.

Side-by-side

Optimization	Cost saved	Quality risk	Validation effort	Ship week
Prompt caching	20-40% of LLM	none	shadow A/B	1
Top-k reduction	10-25% of LLM	low (often improves)	faithfulness curve	2
Vector quantization	50-75% of vector DB	low (reranker recovers)	recall@k	3-4
Generator routing	30-50% of LLM	real, bounded	per-tier eval + A/B	month 2
Semantic answer cache	5-20% of pipeline	staleness	hit ratio + freshness	month 2
Self-hosted Llama 4	3-5x on volume tier	model quality gap	4-6 week pilot	month 3-4

Real-world examples

Real products, models, and research that use this idea.

Anthropic's prompt caching docs show 90% cost reduction on cached tokens, with a 5-minute default cache window suitable for RAG system prompts.
OpenAI's prompt caching auto-caches repeated prefixes and is documented to cut cost 50% on cached portions.
Pinecone serverless tiering moves cold vectors to cheaper storage automatically, often cutting vector DB cost 60-80% for large corpora.
Anthropic Claude 4.7 contextual retrieval reduces the effective top-k needed because each chunk carries its own context, often dropping prompt size 30-40%.
Self-hosted Llama 4 on B200 clusters with vLLM achieves 3-5x cost per token savings vs hosted APIs at sustained 100K+ TPS.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build the generator router classifier, and how do you tune its threshold against the 1-point faithfulness budget?

QHow do you decide when to switch from API to self-hosted Llama 4?

QWhat is the right TTL for semantic answer caching, and how do you invalidate on corpus updates?

QHow does prompt caching interact with conversational multi-turn RAG?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Jumping to fixes without profiling. If you do not know the cost breakdown, you optimize the wrong layer. The LLM dominates; embedding cost is rarely worth chasing first.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.