Production RAG cost is $1M/month. Quality budget: faithfulness can drop at most 1 point. Walk through the highest leverage cost optimizations in priority order. For each, name the expected cost reduction, the quality risk, and how you'd validate the change before rolling it out.
Profile first; the LLM eats 50-65% of cost. Ship in order: prompt caching, top-k reduction, vector quantization, generator routing, semantic cache, self-hosted Llama. Gate each change on the eval set.
Think of your RAG bill like a household budget. Before you cut anything, you figure out where the money actually goes. In most homes the rent or mortgage is the biggest line item; in RAG, it is the smart-AI call that writes the answer. Once you know that, you attack the biggest bills first, but you start with the changes that cannot hurt quality: turn on the discount card the AI provider already gives you, send shorter shopping lists, and shrink your storage closet. Only then do you try the riskier moves, like using a cheaper helper for easy questions or running your own AI on your own machines. After each change, you check that the answers are still as honest as before. If they slip more than a tiny bit, you back the change out.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A $1M/month RAG bill is a real budget conversation, and the CFO's 50% cut with a 1-point faithfulness ceiling is a realistic ask if you sequence the optimizations correctly. The trap is to jump to fixes without profiling, or to ship the highest savings move first and burn weeks of eval risk for a smaller marginal win.
The senior signal is doing the profile, ordering by leverage divided by risk, and tying every change to a measurable check on the golden eval set. This deep dive walks the cost profile, the six highest leverage levers, the order to ship them, and the combined effect within a quarter.
Profile the bill before touching anything
A typical $1M/month production RAG stack breaks down roughly as follows.
- LLM generation: 50-65%, or $500K-650K/month. Frontier-model token cost on every query.
- Vector DB: 10-20%, or $100K-200K/month. Storage and read capacity for the index.
- Reranker: 5-15%, or $50K-150K/month. Cross-encoder calls per query.
- Embedding API: 5-15%, or $50K-150K/month. Ingest plus query-time embedding.
- Infra: 5-10%, or $50K-100K/month. Serving, monitoring, networking.
The LLM dominates. Roughly two-thirds of the bill, depending on whether you use a frontier model on every query or already mix tiers. Cost engineering follows the dominant line; chasing the embedding API saves at most 15% even if you cut it to zero, while a 40% cut on the LLM saves 25%+ of the total.
The second observation is that the optimizations interact. Prompt caching reduces LLM token cost; top-k reduction reduces input tokens to the LLM, which reduces what is left for caching to discount. Order the changes so you measure each in isolation against the previous baseline, not against a moving target.
Profiling tools: most cloud providers expose cost per service dashboards; LLM vendors expose per-model token usage; you wire your own cost per query telemetry into the serving layer. Aim for daily resolution before you ship the first optimization.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Optimization | Cost saved | Quality risk | Validation effort | Ship week |
|---|---|---|---|---|
| Prompt caching | 20-40% of LLM | none | shadow A/B | 1 |
| Top-k reduction | 10-25% of LLM | low (often improves) | faithfulness curve | 2 |
| Vector quantization | 50-75% of vector DB | low (reranker recovers) | recall@k | 3-4 |
| Generator routing | 30-50% of LLM | real, bounded | per-tier eval + A/B | month 2 |
| Semantic answer cache | 5-20% of pipeline | staleness | hit ratio + freshness | month 2 |
| Self-hosted Llama 4 | 3-5x on volume tier | model quality gap | 4-6 week pilot | month 3-4 |
Real products, models, and research that use this idea.
- Anthropic's prompt caching docs show 90% cost reduction on cached tokens, with a 5-minute default cache window suitable for RAG system prompts.
- OpenAI's prompt caching auto-caches repeated prefixes and is documented to cut cost 50% on cached portions.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build the generator router classifier, and how do you tune its threshold against the 1-point faithfulness budget?
Train a small XGBoost or distilled BERT model on past traffic, labels from faithfulness scores on each tier. Tune the routing threshold to maximize routing to small model rate subject to the slice level faithfulness eval staying within 1 point. Re-tune monthly as traffic shifts.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Jumping to fixes without profiling. If you do not know the cost breakdown, you optimize the wrong layer. The LLM dominates; embedding cost is rarely worth chasing first.
60 second bullets to scan on the way to the call.
Why profiling comes before any optimization
The rough cost breakdown of a typical $1M/month RAG stack
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.