Which statement best explains why prefill and decode hit different bottlenecks on the same GPU?

An LLM request has two phases. **Prefill** reads your whole prompt at once and produces the first output token. Decode then generates the rest of the output one token at a time. Both phases run the same model on the same GPU, yet they hit different limits. The reason is how many useful calculations the GPU does for each byte it reads from memory. That ratio is called arithmetic intensity. In prefill, the model loads each weight once and uses it on many prompt tokens in parallel. One memory read serves lots of math, so the GPU's compute units are the limit. We call this compute-bound. In decode, the model produces just one token per step, but it still has to read every weight from memory to do it. Now one giant memory read serves a tiny amount of math. The GPU sits waiting on memory, so this phase is memory bandwidth bound. This is why decode usually dominates total latency for long answers, and why most serving tricks like batching, KV cache management, and speculative decoding target decode rather than prefill.

Both phases run identical weights at identical precision on the same GPU. The difference is arithmetic intensity, meaning FLOPs done per byte read from HBM. Prefill processes the full prompt in one parallel pass. Each weight matrix is read once and reused across every prompt token. That pushes intensity high, roughly 10 to 100 FLOP per byte. The workload sits under the compute roof and is compute-bound. Prefill therefore sets time to first token, which scales with input length. Decode generates one token per step. To produce that single token, the GPU must still stream the entire weight set from HBM. Intensity collapses toward 1 FLOP per byte. The GPU starves on memory bandwidth, so decode is memory-bound. Decode sets time per output token, and it dominates total latency whenever outputs are long. The discriminator is parallelism over the sequence dimension, not parameter count or dtype. This is why batching helps decode so much. Stacking many requests lets one weight read serve many tokens, raising intensity and reclaiming wasted bandwidth.