On the roofline, which intervention moves an LLM decode workload most directly toward higher peak throughput?

The **roofline model** is a one-chart way to ask: is my workload limited by how fast the chip can compute, or by how fast it can read data from memory? The horizontal axis is arithmetic intensity, the FLOPs done per byte moved. The vertical axis is achievable throughput. The chart has two parts. A sloped line on the left is the memory-bandwidth ceiling. A flat line on the right is the peak-compute ceiling. Where they meet is the ridge point. LLM decode, which generates one token at a time, does very little math per byte of weights it reads. So it lands far to the left, on the bandwidth slope. The compute units sit mostly idle, waiting on memory. When you are on the slope, the only way up is to move fewer bytes per unit of work, or to read those bytes faster. INT4 quantization shrinks each weight from 16 bits to 4 bits, so you stream a quarter of the bytes. That moves you up the slope. Cutting FLOPs by pruning does not help here, because FLOPs were never the bottleneck. That is the trap the question sets, and the whole point of the roofline framing.

The **roofline model** plots achievable throughput against arithmetic intensity, the ratio of FLOPs done per byte read from memory. It has two ceilings. A sloped line is the memory-bandwidth limit. A flat line is the peak-compute limit. The two meet at the ridge point. The ridge sits at peak FLOPs divided by peak bandwidth. Workloads left of the ridge are memory bound. Workloads right of it are compute bound. LLM prefill processes the whole prompt in parallel, so it has high intensity and lands right of the ridge, compute bound. Decode emits one token per step and reuses the same weights for tiny work, so it has very low intensity and lands far left, bandwidth bound. For a bandwidth-bound workload, throughput equals bandwidth times intensity. You raise it two ways. Lift the bandwidth roof with faster memory. Or raise intensity by moving fewer bytes per FLOP. INT4 quantization cuts bytes per weight by four. The FLOP count is unchanged, but each FLOP now hides behind a quarter of the memory traffic. That raises effective intensity and slides the kernel up the slope. Pruning FLOPs does the opposite of useful here. It trims the axis you are not limited by, so decode latency barely moves.