Cost Optimization

Which levers cut RAG serving cost without gutting answer quality?

Cutting RAG prompt tokens without dropping whole retrieved chunks: how does context compression work?

Select every lever that reduces the cost of a chatty multi-agent workflow

PremiumRunning LLM-as-judge on your…

You have 500KB of relevant documentation that exceeds the 200K-token context window of your production model. How should you choose between context stuffing (fit what you can) and RAG retrieval?

Your production RAG costs $1M/month. The CFO wants this cut in half with a max 1 point faithfulness regression. What highest leverage cost optimizations do you deploy, in priority order?

Your team is paying $40k/month on LLM tokens: most calls share a long system prompt + 8k tokens of few-shot examples. How does prompt caching help and what's the expected savings?

In a production LLM app with a stable system prompt + long retrieved context + variable user query, which part of the prompt becomes cacheable via Anthropic/OpenAI prompt caching?

What's the highest leverage cost optimization for a production LLM app spending $10k+/month on tokens, BEFORE prompt level token trimming?

Estimate the per call cost of a typical RAG chatbot using GPT-4o-mini.