Click any words you think contain an error. Click again to unmark.
Output costs several times more than input because decode runs one sequential bandwidth-bound forward pass per token, while prefill processes the whole prompt in one parallel compute-bound pass.
Imagine a bakery. Reading the customer's whole order at once is fast: one glance at the slip and you know everything they want. That is the input prompt. Now you have to bake each cake one at a time, and before every single cake you must walk to the back room and haul out the entire recipe book. That trip to the back room is the slow part, and you repeat it for every cake. That is generating output tokens: each one drags the whole model out of memory again. So reading the order is cheap and amortized, while producing each new item is repeated heavy lifting. That is why providers charge more per output token than per input token, not because of any random dice roll.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The price gap between input and output tokens is one of the cleanest tests of whether someone actually understands LLM serving. The passage in this question states the conclusion correctly, output costs more than input, but every reason it gives is wrong. That makes it a perfect spot the error exercise: the surface claim is right, so a shallow reader nods along, while the mechanism underneath is backwards. An interviewer uses exactly this shape of prompt to find out whether you reason from a real model of the hardware or from a memorized slogan.
The three errors are blaming output cost on random sampling, claiming input tokens skip the forward pass, and asserting that GPU FLOPs are the bottleneck in both phases. To dismantle all three you need one core idea: inference runs in two phases, prefill and decode, and they saturate different hardware resources. Once that idea is in place, each false claim falls over on its own, and you can also explain why every serious serving optimization in production exists.
This deep dive builds that mental model from the ground up. We look at what actually happens to an input token versus an output token, which resource each phase exhausts, why that produces a several-fold price difference, and how batching and prompt caching confirm the story. By the end you should be able to explain the pricing differential on a whiteboard without ever mentioning a random number generator, and to back of envelope the numbers that make the ratio land where it does.
Error one: sampling is not the cost driver
The passage says each output token is expensive because it must be sampled with a random number generator. This is the kind of claim that sounds technical but collapses under a single number. Drawing one token from a probability distribution over the vocabulary is a softmax over the logits followed by a draw, perhaps with top-k or top-p filtering first. Even over a vocabulary of 100,000 or more entries, that is a few microseconds of work on the GPU.
The real cost of producing one output token is the forward pass that comes before the sample. The model must push the new token's hidden state through every layer, and at each layer it reads the layer's weight matrices out of HBM. For a frontier model that is tens of gigabytes of weights streamed per token, which lands in the tens of milliseconds. The sampling step is three to four orders of magnitude smaller.
There is a clean way to falsify the RNG claim. Switch to greedy decoding, which removes the random draw entirely and just takes the argmax. Throughput barely moves, because the cost was never in the draw. If sampling were the driver, greedy decoding would be dramatically cheaper, and it is not.
So the sampling kernel is rounding error. The forward pass is the whole bill. Any answer that fixes on the RNG has confused the cheap final step with the expensive computation that feeds it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI and Anthropic both publish output token prices several times higher than input prices; Claude Opus 4.7 and GPT-5.5 list output at roughly 4 to 5 times input.
- vLLM continuous batching exists specifically to raise decode throughput, the bandwidth-bound phase, by amortizing weight reads across many concurrent requests.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does batching help decode so much more than prefill?
Decode is bandwidth-bound with tiny arithmetic intensity, so it leaves the GPU math units idle. Batching many requests reuses one weight read across all of them, raising intensity toward the compute roofline. Prefill is already compute-bound, so it has far less headroom to gain.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Attributing output cost to RNG sampling, or claiming input tokens skip the forward pass. Sampling is microseconds; input runs a real, parallel prefill pass.
60 second bullets to scan on the way to the call.
Why prefill is cheap per token and decode is expensive per token
Which hardware resource saturates in prefill versus decode
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.