Apply the roofline model to LLM inference and identify where decode and prefill sit on it.

The **roofline model** is one chart that tells you whether a workload is limited by how fast the chip computes or by how fast it can move data from memory. The horizontal axis is arithmetic intensity, the FLOPs done per byte read from HBM. The vertical axis is attainable throughput in FLOP/s. The chart has two parts. A sloped line rising from the origin is the memory-bandwidth limit: with low intensity, every byte you fetch only buys a little compute, so bandwidth caps you. A flat ceiling is the peak compute the chip can sustain. Where they meet is the ridge point. On an H100 with about 3 TB/s of HBM and 989 TFLOPs of FP16, the ridge sits near 330 FLOPs per byte. Anything left of that is memory-bound; anything right of it is compute-bound. Prefill processes the whole prompt at once, so it reuses weights across many tokens and lands at moderate intensity. Decode generates one token at a time, reading all the weights to produce a single token, so its intensity is roughly one. Decode sits far down on the slope, wasting the compute units. Batching is the fix. Serving many requests together reuses each weight read across all of them, raising intensity and sliding decode rightward toward the compute roof.

The **roofline model** answers one question. Is this kernel limited by compute or by memory bandwidth? The ridge point is where the two limits meet: $$\text{AI}^* = \frac{\text{peak compute}}{\text{bandwidth}} = \frac{989 \times 10^{12}}{3 \times 10^{12}} \approx 330 \ \text{FLOP/byte}$$ Utilization is capped by the lower roof. Below the ridge you are bandwidth-bound. Above it you are compute-bound. Attainable throughput equals the minimum of peak compute and bandwidth times intensity. Prefill processes the whole prompt in parallel. It reuses each weight across many tokens. So its intensity lands roughly between 10 and 100, near or below the ridge, depending on sequence length. Decode at batch 1 reads every weight to emit a single token. Its intensity is about 1. That is far left on the slope, reaching only about 0.3 percent of peak compute. Decode is firmly bandwidth-bound. Batching is the lever. Running batch size B reuses each weight read across B requests. So intensity scales roughly with B, and decode crosses into compute-bound near a batch of 330. The model tells you to attack bytes, not FLOPs. Raise intensity through batching, or cut bytes through quantization and grouped-query attention.