Claude Sonnet (illustrative rates for this calculation): - input tokens: USD 3 per 1M - output tokens: USD 15 per 1M - cached input tokens: USD 0.30 per 1M A single API call has: - 10,000 cached input tokens (system prompt + tool schemas, cache HIT) - 2,000 new input tokens (this query's retrieved chunks + user question) - 600 output tokens (the assistant's reply) Compute the total cost of this single call in US cents (¢). Round to two decimal places. Then state which component (input / cached / output) dominates the cost.
Cost is a three-line sum: cached input plus new input plus output. Output tokens dominate because the output rate is 5x input and 50x cached input, even with far fewer tokens.
Imagine a print shop with three price tiers. Photocopying a page you already printed last week is almost free. Printing a fresh page costs a bit more. But writing a brand-new page by hand, word by word, is the expensive part. An API call works the same way. The huge system prompt you reuse every time is the cheap photocopy. This query's fresh context is the mid-price print. The model's reply is the hand-written page, generated one token at a time, billed at the top rate. So even a short reply can cost more than a giant cached prompt. To spend less, you do not trim the prompt first. You make the model say less.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Per-call cost is the most practical math in inference optimization, because it links engineering choices to the bill. Every product decision, from how long a system prompt to ship to how verbose an agent should be, eventually shows up as a number on a billing dashboard. Splitting that number into its parts shows which part to reduce.
The common error here is volume bias. The call has 10,000 cached tokens, 2,000 fresh input tokens, and only 600 output tokens. The instinct is to assume the 10,000-token bucket dominates because it is by far the largest count. It does not. Pricing is not uniform across token types, and the smallest bucket here carries the highest rate by a wide margin.
Ranking the line items by cost gives the optimization order: on most real workloads, shortening the model's output saves more than trimming the prompt.
Three buckets, three rates
LLM APIs do not bill tokens at a single price. They expose at least three tiers, and the gap between them is large.
- Cached input tokens, here USD 0.30 per million. These are tokens the provider already processed on a previous call and kept, so re-processing is cheap.
- Fresh input tokens, here USD 3 per million. These are new prompt tokens the model must run through a full prefill pass.
- Output tokens, here USD 15 per million. These are tokens the model generates, one at a time, during decode.
The rates span a 50 to 1 range from cheapest to most expensive. A correct cost estimate needs the buckets kept separate, each billed at its own rate. Blending them into one average rate, or worse, applying the input rate to everything, gives a wrong answer and hides the term that actually controls spend.
On a real request, each bucket holds different content. The cached bucket is the stable part of the prompt: a long system prompt, tool and function schemas, retrieval instructions, and few-shot examples that never change between calls. The fresh input bucket is whatever is unique to this turn: the user's question and the retrieved chunks a RAG pipeline just fetched. The output bucket is the model's reply. Put every token in the right bucket before multiplying, because a token in the wrong bucket can be off by 10x or 50x in price.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude prompt caching cuts cached input to roughly a tenth of the standard input rate, the discount modeled in this calculation.
- OpenAI's automatic prompt caching applies a similar cached-input discount on GPT-5.5 (0.1x the input rate), making long static system prompts cheap to reuse.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the output rate so much higher than the input rate?
Tie it to the two inference phases. Input is processed in one parallel prefill pass, compute-bound and efficient. Output is decoded one token at a time, each step streaming the weights and the full KV cache from HBM. Memory-bandwidth-bound sequential work is expensive, and a higher output rate is consistent with that.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating all tokens as one price, or assuming the 10,000-token prompt must dominate because it is biggest. The output line wins on price per token, not on count.
60 second bullets to scan on the way to the call.
The three token buckets and the distinct rate each one carries
How to compute one line item as count times rate over one million
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.