'Prompt caching cuts our bill by 90 percent across the board', what's overclaimed?

Spot the error·Medium·4.0 · 0·~2 min·Asked atDataiku Elevenlabs·Relevant atAnthropic OpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

**LLM APIs charge for input and output separately.** Input tokens are what you send in: the system prompt plus the user message. Output tokens are what the model writes back. Output usually costs about five times as much as input per token, and up to about eight times on some models. Prompt caching discounts only the input side. The provider stores the work it did for the leading tokens of your prompt. The next time you send exactly the same leading tokens, it charges a much lower rate for them. Output is never cached, because each answer is new. Every generated token costs the full output rate. The app sends about 2,200 input tokens and gets 3,000 output tokens back. A 90 percent discount on the 2,000 cached input tokens is a real saving, but it is a small part of the bill. The 3,000 output tokens at five times the input price cost far more than all the input. So the bill drops by about 10 percent, not 90 percent. Read the price sheet. Apply the cache discount only to the cached input tokens, then recompute the total.

**Two errors compound.** First, the cache discount applies to input tokens only. Output is billed at the full rate. Second, output usually costs about five times the input rate (up to about eight times on some models). So output dominates the invoice on any generation-heavy workload. Work the scenario in input-token equivalents. Input is 2,200 tokens. The 2,000 cached tokens at a 90% discount cost the same as 200 full-price tokens. Add the 200 user tokens for 400 input equivalents. Output is 3,000 tokens at 5x, or 15,000 equivalents. The cached total is 15,400. Without caching, the call costs 2,200 + 15,000 = 17,200. The saving is 1,800 / 17,200, about 10.5 percent. The 90% headline is almost nine times too high. The cost model has three terms: (cached_input x d x input_price) + (uncached_input x input_price) + (output_tokens x output_price), where d is the cached-read multiplier. Caching changes only the first term. On output-heavy workloads (writing assistants, code generation, chat), caching saves single-digit to low double-digit percentages. On input-heavy workloads (long-context RAG with short answers, agent loops with large tool schemas and short tool calls), input is most of the bill and the saving approaches but never reaches the headline. The ceiling on bill reduction is the cached share of the bill times (1 - d). If cached input is fifty percent of the bill, the most caching can save is 45 percent. Reprice on the real workload before promising a number.

TL;DR

Prompt caching discounts INPUT tokens only, not output.

Memory aid

Easy to grasp

Imagine a bakery that gives you ninety percent off flour but full price on the cake itself. If your order is mostly cake with a little flour, the discount barely shows up on the receipt. Prompt caching is the flour discount. It cuts the cost of the long, repeated instructions you send in, but it does nothing for the long answer the model writes back. When the model writes a long essay, that essay is the cake, and you pay full price for every word. The total bill drops a little, not a lot.

Key concepts

Prompt caching Cost modeling Input vs output pricing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Prompt caching saves a lot on input-heavy workloads and little on output-heavy ones. The claim takes a real provider feature, the ninety-percent input-token discount on cache hits, and applies it to the whole invoice. On this scenario the prediction is almost nine times too high, because it applies a per-token input discount to a bill that is mostly output.

The scenario is a long-form writing assistant: about 2,200 input tokens (2,000 system, 200 user) producing about 3,000 output tokens of essay. The team is right that the system prompt is stable across requests and that caching it should help. The error is the next step, the assumption that the per-token discount on cached input carries through to the whole bill. It does not, for two reasons that compound.

A simple cost formula built from the price sheet gives a quick estimate of the bill effect of caching for any workload. It also points to workload changes that recover more of the discount.

How LLM pricing works

Every major provider charges separately for input and output tokens, and the two prices are very different. Every current Claude model lists output at exactly 5x input. OpenAI's current flagships are also 5x, GPT-5.5 and GPT-5.4 are 6x, and the GPT-5 family is 8x. Google's Gemini models range from 5x to about 8x. The factor varies by provider and model, but output always costs several times more than input.

The price difference reflects the underlying serving cost. Input is processed in the prefill phase, which runs all the input tokens through the model in one parallel forward pass. That phase is compute-bound on current GPUs and amortizes well across batched requests, especially when prompts share a prefix. Output is produced in the decode phase, which generates tokens one per forward pass and is memory-bandwidth-bound. Each output token costs the provider much more GPU time than each input token.

Prompt caching adds a third price. On a cache hit, the provider skips the prefill work for the cached portion of input and charges a much lower rate. Anthropic charges 0.1x the input rate for cache reads on most models (0.05x on Claude Opus 5.5 and Sonnet 5.5), and 1.25x for a 5-minute cache write. OpenAI charges 0.1x for cached input on GPT-5.x models. GPT-5.6 and later also charge 1.25x for cache writes, while GPT-5.5 and earlier have no write charge. The exact factor matters for the math, but every provider works the same way: the discount applies to input only, only to the cached prefix, and not at all to output.

The original argument misses this. Output is the expensive part of the bill, and caching does not change it. On any workload with substantial output, the headline discount cannot carry through to the invoice.

The full cost formula

Applying the formula to the scenario

Workloads where caching saves the most

Cost-engineering implications

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Claude Sonnet 5.5 lists output at 5x input and cache reads at 0.05x input. On this essay workload that still cuts the bill by only about 11%.
OpenAI's GPT-5 family lists output at 8x input and cached input at 0.1x. On this essay workload the bill drops by only about 7%.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the maximum possible bill reduction from prompt caching alone, even with a perfect hit rate?

The cap is the input share of the bill multiplied by (1 - d). If output is 80% of the bill, the input share is 20%, and even a perfect 90% input discount cuts only 18% of the total. The output side cannot be touched, so saving has a hard ceiling set by the workload mix.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Multiplying a per-token input discount by the whole invoice and predicting a 90% bill cut. Output tokens carry the bulk of cost on generation-heavy workloads and the cache does not touch them.

60 second bullets to scan on the way to the call.

How input and output tokens are billed separately, with output usually about 5x input
Why prompt caching discounts only input tokens, never output

Primary sources. Browse if you want the original framing.