The **roofline model** is a one-chart way to ask: is my workload limited by how fast the chip can compute, or by how fast it can read data from memory? The horizontal axis is arithmetic intensity, the FLOPs done per byte moved. The vertical axis is achievable throughput.
The chart has two parts. A sloped line on the left is the memory-bandwidth ceiling. A flat line on the right is the peak-compute ceiling. Where they meet is the ridge point.
LLM decode, which generates one token at a time, does very little math per byte of weights it reads. So it lands far to the left, on the bandwidth slope. The compute units sit mostly idle, waiting on memory.
When you are on the slope, the only way up is to move fewer bytes per unit of work, or to read those bytes faster. INT4 quantization shrinks each weight from 16 bits to 4 bits, so you stream a quarter of the bytes. That moves you up the slope.
Cutting FLOPs by pruning does not help here, because FLOPs were never the bottleneck. That is the trap the question sets, and the whole point of the roofline framing.
The **roofline model** plots achievable throughput against arithmetic intensity, the ratio of FLOPs done per byte read from memory. It has two ceilings. A sloped line is the memory-bandwidth limit. A flat line is the peak-compute limit.
The two meet at the ridge point. The ridge sits at peak FLOPs divided by peak bandwidth. Workloads left of the ridge are memory bound. Workloads right of it are compute bound.
LLM prefill processes the whole prompt in parallel, so it has high intensity and lands right of the ridge, compute bound. Decode emits one token per step and reuses the same weights for tiny work, so it has very low intensity and lands far left, bandwidth bound.
For a bandwidth-bound workload, throughput equals bandwidth times intensity. You raise it two ways. Lift the bandwidth roof with faster memory. Or raise intensity by moving fewer bytes per FLOP.
INT4 quantization cuts bytes per weight by four. The FLOP count is unchanged, but each FLOP now hides behind a quarter of the memory traffic. That raises effective intensity and slides the kernel up the slope. Pruning FLOPs does the opposite of useful here. It trims the axis you are not limited by, so decode latency barely moves.
Quick reference
TL;DR
Decode sits on the roofline's memory-bandwidth slope, so the lever that helps is one that cuts bytes moved per FLOP. INT4 quantization does that; pruning FLOPs does not.
Picture a kitchen where the chef is lightning fast, but ingredients arrive on one slow conveyor belt. The chef finishes each dish instantly, then stands around waiting for the belt. Making the chef faster does nothing, because the belt is the bottleneck. Cutting the recipe's steps does nothing either, because the chef was never the holdup. The only thing that helps is shrinking each ingredient so more fits on the belt per trip. Squeeze every ingredient to a quarter of its size and the same belt now delivers four times as many per second, so the chef finally has enough to work with. The roofline is just a chart that tells you whether you are chef-limited or belt-limited.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Reasoning about prefill latency on long prompts rather than token by token decode Prefill is compute bound and already near the flat roof, so bandwidth-side levers like quantization help far less there.
Workloads dominated by kernel launch overhead or host to device transfer The roofline assumes the kernel runs long enough to hit a roof; tiny operations are bound by overhead, not bandwidth or compute.
Comparing accuracy tradeoffs of low-bit quantization rather than throughput The roofline says nothing about quality; INT4 may climb the slope yet degrade outputs unacceptably for the task.
Pipeline limited or network limited multi-node serving When interconnect or scheduling dominates, the single-chip roofline mispredicts the real bottleneck entirely.
Quick scan · 2–4 min
Everything important, quickly.
4 min: roofline axes + ridge point formula + why decode is left of ridge (memory bound) and prefill right + which levers raise arithmetic intensity (quantization, batching) versus which only cut FLOPs.
Real-world examples
Real products, models, and research that use this idea.
1NVIDIA's H100 (989 TFLOP/s bf16, 3.35 TB/s HBM3) gives a bf16 ridge near 295 FLOPs per byte; LLM decode sits far left of it.
2vLLM uses continuous batching to raise decode arithmetic intensity, amortizing one weight read across many concurrent requests toward the ridge.
3DeepSeek V4 and Llama 4 ship INT4 and FP8 weight quantization specifically to cut the bytes streamed per decode step.
4TensorRT-LLM (NVIDIA) exposes roofline-style profiling so engineers can confirm a decode kernel is HBM bound before tuning it.
5FlashAttention raises the attention kernel's arithmetic intensity by tiling, moving it rightward toward the compute roof on H100 and B200.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow exactly does batching move a decode kernel along the roofline?
Sign in or go Pro to see the approach for this follow-up.
QWhy does INT4 quantization raise arithmetic intensity but FP16 to BF16 does not?
Sign in or go Pro to see the approach for this follow-up.
QWhere does the KV cache read fit on the decode roofline?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming that cutting FLOPs (pruning, smaller matmuls) speeds up decode. Decode is bandwidth bound, so only cutting bytes moved per token helps; fewer FLOPs leave the bottleneck untouched.