Why are output tokens typically 3-5× more expensive than input tokens?

A request has two phases. **Prefill** reads your prompt. The model processes every input token in one parallel forward pass, computing all their keys and values at once. Because many tokens go through the GPU together, the matmuls are large and the hardware stays busy. This phase is compute-bound, and it is cheap per token. Decode writes the reply. The model emits one token, feeds it back in, emits the next, and so on. Each step is a separate forward pass that produces a single token. To produce that one token, the GPU must read all the model weights out of memory again. This is the catch. Reading tens of gigabytes of weights for just one token wastes most of the hardware's compute. The bottleneck is memory bandwidth, not arithmetic, so decode is much slower per token than prefill. Slower per token means fewer tokens per second per GPU, which means higher cost to serve each one. Providers pass that cost through, so output is priced roughly three to five times input. The ratio is physics, not margin.

Pricing tracks how many tokens a GPU can serve per second in each phase. **Prefill** processes the whole prompt in one parallel pass. Many tokens share a single read of the model weights from memory. Arithmetic intensity is high, the GPU hits its compute roof, and per-token cost is low. Decode is the opposite. The model generates output one token at a time. Each step reads the entire weight matrix from HBM to produce a single token. That makes decode memory-bandwidth bound, not compute bound. A single model instance tops out around 80 to 150 tokens per second regardless of how many FLOPs the chip can do. This throughput gap is the whole story. Prefill amortizes one weight read over thousands of prompt tokens. Decode amortizes one weight read over a small batch of concurrent users. So output costs more hardware time per token served. Providers price to recover GPU cost over realized throughput. The 3-5x ratio is the prefill to decode throughput gap showing up on the invoice. It reflects hardware physics, not an arbitrary markup.