Spot the errors in this explanation of input vs output pricing

**LLM inference** has two phases, and they have very different costs. The first is prefill: the model reads your whole prompt at once. The second is decode: the model writes its answer one token at a time. Prefill is cheap per token because all the input tokens go through the model together in a single parallel pass. The expensive part of any forward pass is reading the model weights out of GPU memory, and prefill pays that cost once while processing hundreds or thousands of input positions at the same time. Decode is expensive per token because it is sequential. To produce each new output token, the model runs a full forward pass and reads the weights again. There is no batching across positions inside one request, so you pay the weight-read cost over and over. The sampling step, picking the actual token from the probabilities, takes microseconds and is irrelevant to cost. The passage blaming a random number generator is simply wrong. Output is several times pricier than input because each output token triggers its own memory-bound pass, not because of any sampling kernel.

The passage gets the direction right but the reasons wrong. Output does cost more than input. The cause is not RNG sampling, which takes microseconds and is negligible. The real story is the two phases of inference. Prefill processes the entire prompt in one parallel forward pass. It is compute-bound and saturates the GPU math units, so cost per input token is low. Decode generates output one token at a time. Each step is a full forward pass that must stream the model weights from HBM, so it is bandwidth-bound and runs at low throughput per GPU. The passage also claims input tokens are merely read from memory with no forward pass. That is false. Input tokens go through a real prefill pass. They are just cheaper per token because the weight read is amortized across many positions at once. The passage finally says the bottleneck is FLOPs in both phases. That is wrong too. Prefill is compute-bound and decode is bandwidth-bound. That asymmetry, not a sampling kernel, is what justifies the 3 to 5 times markup on output.