Spot the errors in this explanation of input vs output pricing

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Output costs several times more than input because decode runs one sequential bandwidth-bound forward pass per token, while prefill processes the whole prompt in one parallel compute-bound pass.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a bakery. Reading the customer's whole order at once is fast: one glance at the slip and you know everything they want. That is the input prompt. Now you have to bake each cake one at a time, and before every single cake you must walk to the back room and haul out the entire recipe book. That trip to the back room is the slow part, and you repeat it for every cake. That is generating output tokens: each one drags the whole model out of memory again. So reading the order is cheap and amortized, while producing each new item is repeated heavy lifting. That is why providers charge more per output token than per input token, not because of any random dice roll.

Key concepts

Prefill vs decode Token pricingMemory bandwidth bound

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: name all three errors, then explain prefill (parallel, compute-bound) versus decode (sequential, bandwidth-bound) and map throughput to per-token price.

Real-world examples

Real products, models, and research that use this idea.

OpenAI and Anthropic both publish output token prices several times higher than input prices; Claude Opus 4.7 and GPT-5.5 list output at roughly 4 to 5 times input.
vLLM continuous batching exists specifically to raise decode throughput, the bandwidth-bound phase, by amortizing weight reads across many concurrent requests.
Prompt caching from Anthropic and OpenAI discounts repeated input prefixes to about 10 percent because cached prefill skips recomputation, showing input cost is prefill compute, not a memory read.
NVIDIA TensorRT-LLM separates prefill and decode scheduling and uses chunked prefill precisely because the two phases saturate different hardware resources.
DeepSeek V4 disaggregated serving runs prefill and decode on separate GPU pools so each phase runs on hardware tuned to its bottleneck.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does batching help decode so much more than prefill?

Sign in or go Pro to see the approach for this follow-up.

QIf output is bandwidth-bound, why not just buy GPUs with more FLOPs?

Sign in or go Pro to see the approach for this follow-up.

QHow does prompt caching change the input side of the cost formula?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Attributing output cost to RNG sampling, or claiming input tokens skip the forward pass. Sampling is microseconds; input runs a real, parallel prefill pass.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.