Decompose the cost of a single API call into its components and explain which dominates.

Attempt it

Write the cost formula for a single LLM API call given non-cached input tokens, cached input tokens, and output tokens. Explain which term typically dominates on production workloads and what that implies for cost optimization.

Free · 2 AI evals / day

Quick reference

TL;DR

Per-call cost splits into new input, cached input, and output tokens; output tokens are roughly five times pricier per token, so output usually dominates the bill.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of a taxi ride with two meters. The first meter runs while you read out your destination to the driver, that is the input. The second meter runs while the driver narrates the whole route back to you, that is the output. The reading-out meter is cheap and ticks fast in parallel. The narrating meter is several times more expensive per word, because the driver speaks one word at a time and you pay a premium for each. There is even a loyalty discount: if you give the same long address every trip, the company remembers it and charges almost nothing to re-read it. So your bill is mostly driven by how much the driver talks back, not how much you said up front.

Key concepts

Per-call cost math Prompt caching Output length controlAgent-loop cost compounding

Quick scan · 2–4 min

Everything important, quickly.

4 min: write the three-term formula, state the output 5x and cached 0.1x ratios, derive the crossover, then rank caching and output trimming as the levers and flag agent-loop compounding.

Side-by-side

Token type	Phase	Relative rate	Cost lever
New input	Prefill, parallel, compute-bound	1x baseline	Trim only after output is minimized
Cached input	Prefill, served from cache	About 0.1x	Cache every stable prefix
Output	Decode, sequential, bandwidth-bound	About 5x	Shorten responses, the top lever

Real-world examples

Real products, models, and research that use this idea.

Anthropic prices Claude Opus 4.7 output tokens roughly five times its input rate, with prompt caching billed near a tenth of input on cache reads.
OpenAI's GPT-5.5 pricing keeps output several times the input rate and offers automatic prompt caching discounts on repeated prefixes.
Cursor and similar coding agents lean on prefix caching because each agent step re-sends a large growing file and tool context as input.
RAG chat products cache the long fixed system prompt and retrieved boilerplate, leaving output length as the main remaining cost driver per turn.
Gemini 3.1 Pro exposes context caching so high-traffic apps amortize a shared instruction prefix across many calls at a steep discount.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QDerive the exact condition under which output cost overtakes input cost.

Sign in or go Pro to see the approach for this follow-up.

QWhy is output billed several times more than input at the hardware level?

Sign in or go Pro to see the approach for this follow-up.

QHow does cost scale in a multi-step agent loop with N tool calls?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.