Decompose the cost of a single API call into its components and explain which dominates.
Write the cost formula for a single LLM API call given non-cached input tokens, cached input tokens, and output tokens. Explain which term typically dominates on production workloads and what that implies for cost optimization.
Per-call cost splits into new input, cached input, and output tokens; output tokens are roughly five times pricier per token, so output usually dominates the bill.
Think of a taxi ride with two meters. The first meter runs while you read out your destination to the driver, that is the input. The second meter runs while the driver narrates the whole route back to you, that is the output. The reading-out meter is cheap and ticks fast in parallel. The narrating meter is several times more expensive per word, because the driver speaks one word at a time and you pay a premium for each. There is even a loyalty discount: if you give the same long address every trip, the company remembers it and charges almost nothing to re-read it. So your bill is mostly driven by how much the driver talks back, not how much you said up front.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Decomposing a single API call into its cost components is one of the most practical questions in inference optimization, because it directly governs whether a product survives at scale. The difference between a one-cent query and a ten-cent query is the difference between a viable business and one that bleeds money on every request.
The naive mental model is that you pay per token, full stop, so the obvious lever is sending fewer tokens. That model is wrong in a way that costs companies real money. Tokens are not fungible. A call has three distinct line items with rates that differ by more than an order of magnitude across them, and the cheapest-looking term is often the one people obsess over while the expensive term runs unchecked.
This deep dive builds the cost formula from first principles, ties each rate back to the prefill and decode hardware story, derives exactly when output overtakes input, and then ranks the optimization levers in the order an experienced engineer actually applies them. The goal is that you can write the formula on a whiteboard, justify every rate from the hardware, and immediately name the highest-leverage cost fix for a given workload.
The three-term cost formula
Every billed call decomposes into three token categories, each with its own price per million tokens. New input tokens are read fresh during prefill. Cached input tokens are a reused prefix served from a stored state. Output tokens are generated during decode.
The total is a simple rate-weighted sum:
The single most important habit is to keep these three terms separate in your head. A blended per-token average hides the structure that actually drives the bill. The rates are not close to each other, so collapsing them into one number throws away the very information you need to optimize. Treat the call as three meters running at three speeds, not one meter counting tokens.
It helps to anchor the rates on real numbers. A current flagship is commonly priced around three dollars per million input tokens and fifteen per million output, with cache reads near thirty cents per million. That is the five-times and one-tenth structure made concrete. The exact figures drift between vendors and tiers, but the ratios are remarkably stable across providers, because they are dictated by the same underlying hardware rather than by marketing. When you sketch this on a whiteboard, write the three rates first; the rest of the analysis is just plugging your workload's token counts into them.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Token type | Phase | Relative rate | Cost lever |
|---|---|---|---|
| New input | Prefill, parallel, compute-bound | 1x baseline | Trim only after output is minimized |
| Cached input | Prefill, served from cache | About 0.1x | Cache every stable prefix |
| Output | Decode, sequential, bandwidth-bound | About 5x | Shorten responses, the top lever |
Real products, models, and research that use this idea.
- Anthropic prices Claude Opus 4.7 output tokens roughly five times its input rate, with prompt caching billed near a tenth of input on cache reads.
- OpenAI's GPT-5.5 pricing keeps output several times the input rate and offers automatic prompt caching discounts on repeated prefixes.
What an interviewer would ask next. Try answering before peeking at the approach.
QDerive the exact condition under which output cost overtakes input cost.
Set output tokens times output rate above input tokens times input rate. With output rate near five times input rate, this reduces to output tokens exceeding input tokens divided by five. Plug in a typical chat turn to show the bar is low.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the bill tracks total token count, so trimming a long prompt is the big win. Output tokens cost several times more each, so a short response usually matters more than a short prompt.
60 second bullets to scan on the way to the call.
The three priced components of a single call and their rate symbols
The two rate ratios for output and for cached input relative to input
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.