Your team is paying $40k/month on LLM tokens: most calls share a long system prompt + 8k tokens of few-shot examples. How does prompt caching help and what's the expected savings?

Short answer·Medium·4.0 · 0·~3 min·Asked atAnthropic Doordash OpenAI

Attempt it

You're an MLOps engineer reviewing a $40k/month LLM token bill. Most calls share the same system prompt (~500 tokens) + ~8k tokens of stable few-shot examples + ~2k tokens of per query content. Explain how prompt caching works, what specifically becomes cacheable, the expected savings, and at least one operational gotcha that could prevent realizing them.

Free · 2 AI evals / day

Prompt caching reuses the work the provider already did on the stable beginning of your prompt. The provider charges much less for that part on the next call. In this setup, each call has about ten and a half thousand input tokens. Five hundred are system prompt, eight thousand are stable few-shot examples, and two thousand are the per query content. The system prompt and the few-shot examples are the same every call, so they are the cacheable prefix. That is roughly eighty-one percent of the input. Anthropic charges about ten percent of normal input price for cached tokens on most models. So each call costs about 2,850 effective input tokens instead of 10,500. That is around a seventy percent reduction in input cost. If input is about seventy percent of the USD 40k bill (around USD 28k), caching takes that to roughly USD 8k. Output stays at USD 12k. Total bill comes out around USD 20k, saving about USD 20k per month. The main caveats are byte-exact match (the prefix must not change), a short cache window, a small extra charge on the call that writes the cache, and the fact that output is never discounted.

Prompt caching is a prefix memoization mechanism the provider exposes at the API layer. The provider stores the processed state for a byte-identical prefix. Later calls with the same prefix skip the prefill compute and pay a heavily discounted rate on the cached portion. For this workload, the cacheable region is the system prompt plus the eight thousand tokens of stable few-shot examples. That is roughly eight thousand five hundred tokens of the ten thousand five hundred input per call. The remaining two thousand tokens of per query content sit at the tail and are billed at full input rate. Output tokens are never discounted. Anthropic gives a roughly tenfold discount on cached input. Eight thousand five hundred tokens at ten percent plus two thousand at full rate gives about two thousand eight hundred fifty effective input tokens per call. That is about a seventy-three percent input reduction. If input is roughly seventy percent of the USD 40k bill (around USD 28k), caching takes the input portion to about USD 8k. Adding back the USD 12k output, total bill drops to about USD 20k, saving roughly USD 20k per month. The first call in each cache window pays a 1.25x write premium on the prefix, so the real figure depends on hit rate. The main gotcha is byte-exact match. Any dynamic injection in the cacheable region (a session id, a timestamp, a reordered tool list) silently invalidates the cache. The team then realizes none of the projected savings.

TL;DR

Cache the 8.5k stable prefix at one-tenth input price; effective input cost drops about seventy percent, taking the USD 40k bill to roughly USD 20k after caveats.

Memory aid

Easy to grasp

Imagine paying a tutor to read your textbook out loud every time you have a question. Most of the textbook is the same; only the question changes. Prompt caching lets the tutor say 'I remember the textbook, just ask your question.' You still pay full price for the question itself and for the tutor's answer, but the textbook is now nearly free per session. Picture an app paying forty thousand dollars a month, where most of the input is the same long set of examples and rules. For that app, this move alone cuts the bill in half. The catch: the tutor only remembers the textbook if you bring it in the exact same words every time. You also have to ask the next question soon enough.

Key concepts

Prompt Caching Inference Optimization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A $40k monthly LLM bill where most input is shared across calls is the standard setup that prompt caching was built for. The savings are large, the mechanism is well-understood, and the operational gotchas are predictable.

Caching is the right move here. The work that remains is operational discipline rather than mechanism design.

How prefix memoization works at the provider

Every inference call builds an attention KV cache as the model processes the prompt. The KV cache stores keys and values for each token across each layer, and the prefill phase walks the prompt forward to build it. Once the prefill is done, decode generates one token at a time against the cached state.

Prompt caching reuses that prefill work across calls. Open-source servers show how: vLLM's automatic prefix caching hashes blocks of prompt tokens and stores their KV cache against the hash. A later request with the same hash reuses the stored state instead of recomputing it. Hosted APIs expose the same idea at the API layer. The cost savings come from skipping the prefill compute on the cached portion. The model still has to do prefill on the fresh suffix and full decode on the output.

A cache hit needs an exact match on the prefix. Even a single character change breaks the match. This is why byte-exact match is the binding constraint on hit rate. Providers expose either explicit cache markers (Anthropic's cache_control) or automatic caching above a length threshold (OpenAI). In both cases the idea is the same: reuse the work done on an identical prefix.

The savings math for the $40k workload

Byte-stability and the silent failure mode

Provider-specific knobs: markers, TTL, model pinning

Action items and the rollout playbook

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic prices prompt cache reads at 0.1x base input on most Claude models, so a stable system plus few-shot prefix costs about a tenth of its normal price on each hit.
OpenAI caches long prefixes automatically and reports hits as usage.prompt_tokens_details.cached_tokens (Chat Completions) or usage.input_tokens_details.cached_tokens (Responses API), which teams track as the hit-rate metric.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you measure cache hit rate after rollout?

Track usage.prompt_tokens_details.cached_tokens (OpenAI) or cache_read_input_tokens (Anthropic) per call type; expect a high ratio on stable workloads; alert on drops as invalidation regressions.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Counting output tokens in the projected savings; caching is input-side only and output is always billed at full rate.

60 second bullets to scan on the way to the call.

Which tokens are cacheable in a system plus few-shot setup
Anthropic tenfold vs OpenAI model-dependent cached-token discount

Primary sources. Browse if you want the original framing.