Your team is paying $40k/month on LLM tokens: most calls share a long system prompt + 8k tokens of few-shot examples. How does prompt caching help and what's the expected savings?

Prompt caching reuses the work the provider already did on the stable beginning of your prompt. The provider keeps the processed state for the prefix and charges almost nothing for it on the next call. In this setup, each call has about ten and a half thousand input tokens. Five hundred are system prompt, eight thousand are stable few-shot examples, and two thousand are the per-query content. The system prompt and the few-shot examples are the same every call, so they are the cacheable prefix. That is roughly eighty-one percent of the input. Anthropic charges about ten percent of normal input price for cached tokens. So the effective input cost per call drops from ten and a half thousand tokens to about three and a quarter thousand effective tokens. That is around a seventy percent reduction in input cost. If input is about seventy percent of the $40k bill (around $28k), caching takes that to roughly $9k. Output stays at $12k. Total bill comes out around $21k, saving about $19k per month. The main caveats are byte-exact match, a short cache window, and the fact that output is never discounted.

Prompt caching is a prefix-memoization mechanism the provider exposes at the API layer. The processed state for a byte-identical prefix gets stored, and subsequent calls with the same prefix skip the prefill compute, paying a heavily discounted rate on the cached portion. For this workload, the cacheable region is the system prompt plus the eight thousand tokens of stable few-shot examples. That is roughly eight thousand five hundred tokens of the ten thousand five hundred input per call. The remaining two thousand tokens of per-query content sit at the tail and are billed at full input rate. Output tokens are never discounted. Run the math under Anthropic's roughly tenfold discount on cached input. Eight thousand five hundred tokens at ten percent plus two thousand at full rate gives about three thousand two hundred effective input tokens per call. That is a seventy percent input reduction. If input is roughly seventy percent of the $40k bill (around $28k), caching takes the input portion to about $9k. Adding back the $12k output, total bill drops from $40k to about $21k, saving roughly $19k per month. The gotcha worth flagging up front is byte-exact match. Any dynamic injection in the cacheable region (a session id, a timestamp, a reordered tool list) silently invalidates the cache, and the team realizes none of the projected savings.