What is the arithmetic intensity of an LLM decode step and why is it close to 1?

**Arithmetic intensity** is a simple ratio: how many math operations you do for each byte you read from memory. High intensity means the chip is busy computing. Low intensity means the chip is mostly waiting for data. In one decode step the model generates a single new token. To do that it must read every weight once. A model with N parameters in FP16 occupies about 2N bytes, because each parameter is 2 bytes. So the step reads roughly 2N bytes from memory. The math done in that step is one forward pass, which costs about 2N floating point operations. So at batch 1 the intensity is 2N divided by 2N, which equals 1. One operation per byte is extremely low. The fix is batching. If you process B requests together, you still read the weights once, but you do the math B times. The intensity becomes B. Bigger batches keep the chip busy and are the main reason production servers run many requests at once.

**Arithmetic intensity** is FLOPs divided by bytes moved from memory. One decode step generates a single token. To do that the GPU must stream every weight once. A model with N parameters in FP16 is about 2N bytes. The forward pass costs about 2N FLOPs. So the intensity at batch 1 is: $$\text{AI} = \frac{\text{FLOPs}}{\text{bytes}} = \frac{2N}{2N} = 1$$ That is roughly one FLOP per byte. It is tiny. Batching is what rescues it. Across B requests the GPU reads the same weights once but does the matmul B times. FLOPs become 2NB while bytes stay 2N. The 2N cancels, so intensity equals B. Intensity is literally the batch size. Now place this on the roofline. An H100 has a critical intensity near 330 FLOPs per byte, the point where it stops being memory-bound. At batch 1, decode sits at intensity 1, far left of that ridge. It achieves only about 1 over 330, roughly 0.3 percent of peak FLOPs. As batch grows, the workload climbs the bandwidth slope linearly. Around B near 330 it crosses into compute-bound territory. This is the exact analytical reason batching is the biggest throughput lever in decode.