'Prompt caching cuts our bill by 90 percent across the board', what's overclaimed?

Spot the error·Medium·4.0 · 0·~2 min·Asked atDataiku Elevenlabs Mckinsey·Relevant atAnthropic OpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

**LLM APIs charge for two things separately.** Input tokens are what you send in, system prompt plus user message. Output tokens are what the model writes back. They have different prices, and the output price is usually three to five times the input price. Prompt caching only discounts the input side. When you turn on caching, the provider stores the work it did for the leading bytes of your prompt and gives you a deep discount the next time you send exactly those same bytes. The discount is on input. Output is not cached, because every output is unique. Each generated token costs the full output rate. The app in the question sends about 2200 input tokens and generates 3000 output tokens. If caching saves 90 percent on the cached portion of input, that is a real saving but a small slice of the total bill. Output is 3000 tokens at roughly four times the input price, which dwarfs even uncached input. So the realized bill reduction is somewhere around 15 percent, not 90 percent. The rule is simple. Read the price sheet. The cache discount is a per-token input discount; it is not a per-invoice discount. Apply it only to the cached input tokens and recompute the total.

**Two errors compound.** First, the cache discount applies to input tokens only; output is billed at full rate every time. Second, output is typically priced at three to five times the input rate, so output dominates the invoice on any generation-heavy workload. Walk the arithmetic on the scenario. Input is 2200 tokens. Of those, 2000 hit the cache at the 90% discount, equating to 200 paid input-tokens, plus the 200 user tokens at full input price, totaling 400 input-token equivalents. Output is 3000 tokens at full output rate. Assuming output is 4x input, output is worth 12,000 input-token equivalents on the bill. Total bill equivalents: 12,400. Without caching, the same call costs 2200 input-equivalents plus 12,000 output-equivalents = 14,200. So the realized saving is roughly 1800/14,200 = 12-15 percent. The headline 90% figure overshoots reality by a factor of six. The correct mental model has three terms: (cached_input * discount * input_price) + (uncached_input * input_price) + (output_tokens * output_price). The cache discount only multiplies the first term. On output-heavy workloads (writing assistants, code generation, chat completion) output dominates, and caching delivers single-digit to low double digit percent savings. On input-heavy workloads (long-context RAG with short responses, agent loops with massive tool schemas and brief tool outputs) the input share of the invoice is large and the saving approaches but never reaches the headline. The ceiling on bill reduction from caching alone is the input share of the bill. If output is fifty percent of the bill, the maximum possible saving is fifty percent times the cache hit rate times the discount factor, even on a perfect workload. Always reprice on the actual workload profile before promising the headline.

TL;DR

Prompt caching discounts INPUT tokens only, not output.

Memory aid

Easy to grasp

Imagine a bakery that gives you ninety percent off flour but full price on the cake itself. If your order is mostly cake with a little flour, the discount barely shows up on the receipt. Prompt caching is the flour discount. It cuts the cost of the long, repeated instructions you send in, but it does nothing for the long answer the model writes back. When the model writes a three thousand word essay, that essay is the cake, and you pay full price for every word. The total bill drops a little, not a lot.

Key concepts

Prompt caching Cost modeling Input vs output pricing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Prompt caching is genuinely transformative for the right workload, and dramatically oversold for the wrong one. The claim under review takes a real provider feature, the ninety-percent input-token discount on cache hits, and projects it across the whole invoice. The result is a prediction that is wrong by roughly six times on the scenario described, and the source of the error is a single category mistake that recurs in many forms in production cost discussions.

The scenario is a long-form writing assistant: about 2200 input tokens (2000 system, 200 user) producing about 3000 output tokens of essay. The author has correctly noticed that the system prompt is stable across requests and that caching it should help. They have then leapt to the conclusion that the per-token discount on cached input flows through to the whole bill. It does not, for two reasons that compound.

This walkthrough builds the right cost model from the price sheet up, applies it to the scenario, and explains which workload shapes genuinely realize the headline discount and which do not. By the end you should be able to project the bill impact of turning on caching for any new workload on a napkin, and to identify workload redesigns that recover more of the headline figure when it matters.

How LLM pricing actually works

Every major provider in 2026 charges separately for input and output tokens, and the two prices are very different. Anthropic's Claude lineup runs roughly $3 per million input tokens and $15 per million output tokens for the mid-tier model; a five to one ratio. OpenAI's GPT-5 and o4 lineup runs roughly four to one. Google's Gemini lineup is similar. The factor varies by provider and SKU but the direction is consistent: output costs more than input, often by a multiple.

The price difference reflects the underlying serving cost. Input is processed in the prefill phase, which runs all the input tokens through the model in one parallel forward pass. That phase is compute-bound on modern hardware and amortizes well across batched requests, especially when prompts share a prefix. Output is produced in the decode phase, which generates tokens sequentially, one per forward pass, and is memory-bandwidth-bound per request. Decode is the expensive side; input is the cheap side. The price sheet bakes this asymmetry in.

Prompt caching adds a third tier. On a cache hit, the provider skips the prefill work for the cached portion of input and charges a deeply discounted rate. Anthropic charges roughly ten percent of the normal input rate on hits, with a slight write premium on the first call that populates the cache. OpenAI charges roughly fifty percent on hits with no write premium. The exact factor matters for the math, but the structural property is the same on every provider: the discount applies to input only, and only to the prefix-hit portion of input, and not at all to output.

This is the entire structural fact the original argument missed. Output is the expensive part of the bill, caching does not touch it, and on any workload where output is non-trivial the headline discount cannot flow through to the invoice.

The full cost formula

Applying the formula to the scenario

Workloads where caching really does win big

Cost-engineering implications

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic Claude pricing in 2026: input $3/M, output $15/M (5x), cache reads at ~$0.30/M (10x discount on input). A 90% input-token cut buys 12-15% bill cut on output-heavy chat.
OpenAI o4 and GPT-5 pricing show output at 4x input with automatic prompt caching at ~50% discount; realized bill savings on generation-heavy apps land near 5-10%.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the maximum possible bill reduction from prompt caching alone, even with a perfect hit rate?

Cap is the input share of the bill multiplied by (1 - d). If output is 80% of the bill, the input share is 20%, and even a perfect 90% input discount cuts only 18% of the total. The output side cannot be touched, so saving has a hard ceiling set by the workload mix.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Multiplying a per-token input discount by the whole invoice and predicting a 90% bill cut. Output tokens carry the bulk of cost on generation-heavy workloads and the cache does not touch them.

60 second bullets to scan on the way to the call.

How input and output tokens are billed separately, with output typically 3-5x input
Why prompt caching discounts only input tokens, never output

Primary sources. Browse if you want the original framing.