Cost Optimization

Running LLM-as-judge on your full eval set costs $200 per run. The team runs evals 10 times a day during development. How do you cut cost without losing signal?

You have 500KB of relevant documentation that exceeds the 200K-token context window of your production model. How should you choose between context stuffing (fit what you can) and RAG retrieval?

Your production RAG costs $1M/month. The CFO wants this cut in half with a max 1-point faithfulness regression. What highest leverage cost optimizations do you deploy, in priority order?

Your team is paying $40k/month on LLM tokens: most calls share a long system prompt + 8k tokens of few-shot examples. How does prompt caching help and what's the expected savings?

In a production LLM app with a stable system prompt + long retrieved context + variable user query, which part of the prompt becomes cacheable via Anthropic/OpenAI prompt caching?

What's the highest-leverage cost optimization for a production LLM app spending $10k+/month on tokens, BEFORE prompt-level token trimming?

Estimate the per call cost of a typical RAG chatbot using GPT-4o-mini.