Spot the errors in this explanation of input vs output pricing
Click any words you think contain an error. Click again to unmark.
Output costs several times more than input because decode runs one sequential bandwidth-bound forward pass per token, while prefill processes the whole prompt in one parallel compute-bound pass.
Imagine a bakery. Reading the customer's whole order at once is fast: one glance at the slip and you know everything they want. That is the input prompt. Now you have to bake each cake one at a time, and before every single cake you must walk to the back room and haul out the entire recipe book. That trip to the back room is the slow part, and you repeat it for every cake. That is generating output tokens: each one drags the whole model out of memory again. So reading the order is cheap and amortized, while producing each new item is repeated heavy lifting. That is why providers charge more per output token than per input token, not because of any random dice roll.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The price gap between input and output tokens comes from how LLM serving uses the hardware. The passage states the conclusion correctly, output costs more than input, but every reason it gives is wrong. The surface claim is right while the mechanism underneath is backwards.
The three errors are blaming output cost on random sampling, claiming input tokens skip the forward pass, and asserting that GPU FLOPs are the bottleneck in both phases. One idea corrects all three: inference runs in two phases, prefill and decode, and they saturate different hardware resources. The same idea explains why the main serving optimizations exist.
An input token and an output token go through different work, and each phase exhausts a different resource. That difference produces a several-fold price gap, and batching and prompt caching both confirm it. A rough calculation with weight bytes and HBM bandwidth shows where the ratio comes from, with no role for a random number generator.
Error one: sampling is not the cost driver
The passage says each output token is expensive because it must be sampled with a random number generator. The claim sounds technical, but one number disproves it. Drawing one token from a probability distribution over the vocabulary is a softmax over the logits followed by a draw, perhaps with top-k or top-p filtering first. Even over a vocabulary of 100,000 or more entries, that is a few microseconds of work on the GPU.
The real cost of producing one output token is the forward pass that comes before the sample. The model must push the new token's hidden state through every layer, and at each layer it reads the layer's weight matrices out of HBM. For a large model that is tens of gigabytes of weights streamed per token, which lands in the tens of milliseconds. The sampling step is three to four orders of magnitude smaller.
There is a clean way to falsify the RNG claim. Switch to greedy decoding, which removes the random draw entirely and just takes the argmax. Throughput barely moves, because the cost was never in the draw. If sampling were the driver, greedy decoding would be much cheaper, and it is not.
So the sampling kernel is a negligible share of the cost, and the forward pass is nearly all of it. Blaming the RNG confuses the cheap final step with the expensive computation that feeds it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI and Anthropic both publish output token prices several times higher than input prices; Claude Opus 5.5 lists output at 5 times input and GPT-5.5 at 6 times.
- vLLM continuous batching exists specifically to raise decode throughput, the bandwidth-bound phase, by amortizing weight reads across many concurrent requests.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does batching help decode so much more than prefill?
Decode is bandwidth-bound with tiny arithmetic intensity, so it leaves the GPU math units idle. Batching many requests reuses one weight read across all of them, raising intensity toward the compute roofline. Prefill is already compute-bound, so it has far less headroom to gain.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Attributing output cost to RNG sampling, or claiming input tokens skip the forward pass. Sampling is microseconds; input runs a real, parallel prefill pass.
60 second bullets to scan on the way to the call.
Why prefill is cheap per token and decode is expensive per token
Which hardware resource saturates in prefill versus decode
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.