On the roofline, which intervention moves an LLM decode workload most directly toward higher peak throughput?
Decode sits on the roofline's memory-bandwidth slope, so the lever that helps is one that cuts bytes moved per FLOP. INT4 quantization does that; pruning FLOPs does not.
Picture a kitchen where the chef is lightning fast, but ingredients arrive on one slow conveyor belt. The chef finishes each dish instantly, then stands around waiting for the belt. Making the chef faster does nothing, because the belt is the bottleneck. Cutting the recipe's steps does nothing either, because the chef was never the holdup. The only thing that helps is shrinking each ingredient so more fits on the belt per trip. Squeeze every ingredient to a quarter of its size and the same belt now delivers four times as many per second, so the chef finally has enough to work with. The roofline is just a chart that tells you whether you are chef-limited or belt-limited.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The roofline model is the single most useful mental model for reasoning about LLM inference performance, and it is a favourite of senior interviewers precisely because it separates engineers who optimize by instinct from those who optimize by diagnosis. The model answers one question: for a given kernel on a given chip, is performance limited by how fast the hardware can compute, or by how fast it can move data from memory?
The chart has two axes. The horizontal axis is arithmetic intensity, the number of floating-point operations performed per byte read from memory. The vertical axis is achievable throughput in FLOPs per second. The chart's ceiling has two segments: a diagonal line set by peak memory bandwidth, and a horizontal line set by peak compute. Their intersection is the ridge point.
This deep dive builds the model from the ridge formula, places LLM prefill and decode on the chart, and then uses it to score the four interventions in the question. The punchline is that decode lives far to the left on the bandwidth slope, so the only interventions that help are the ones that change the bytes axis, namely quantization and batching, not the ones that change the FLOPs axis, namely pruning.
The ridge point and what the two roofs mean
The diagonal roof says throughput is capped at bandwidth times arithmetic intensity. The flat roof says throughput is capped at the chip's peak FLOP rate. A kernel runs at whichever ceiling is lower for its intensity. The crossover is the ridge point, and it is the heart of the model.
The ridge sits where the two roofs meet:
The units are FLOPs per byte. On an H100, roughly 989 teraFLOPs of bf16 compute divided by about 3.35 terabytes per second of HBM bandwidth gives a ridge near 295 FLOPs per byte. Any kernel whose arithmetic intensity is below that number is memory bound and lives on the slope. Any kernel above it is compute bound and lives under the flat roof.
The ridge number is a property of the hardware, not the workload, which is why the same kernel can be memory bound on one chip and compute bound on another. It also explains why generational hardware shifts matter: peak FLOPs has grown faster than memory bandwidth across GPU generations, so the ridge has crept rightward over time. That means a workload that was comfortably compute bound on an older chip can become memory bound on a newer one, even though nothing about the workload changed. The roofline forces you to recompute the ridge for the exact silicon you are deploying on, rather than carrying over intuitions from a previous generation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- NVIDIA's H100 (989 TFLOP/s bf16, 3.35 TB/s HBM3) gives a bf16 ridge near 295 FLOPs per byte; LLM decode sits far left of it.
- vLLM uses continuous batching to raise decode arithmetic intensity, amortizing one weight read across many concurrent requests toward the ridge.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow exactly does batching move a decode kernel along the roofline?
One weight read serves every request in the batch, so FLOPs scale with batch size while bytes read stay roughly fixed. Arithmetic intensity rises with batch size until the kernel reaches the ridge and becomes compute bound, after which more batching stops helping latency.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming that cutting FLOPs (pruning, smaller matmuls) speeds up decode. Decode is bandwidth bound, so only cutting bytes moved per token helps; fewer FLOPs leave the bottleneck untouched.
60 second bullets to scan on the way to the call.
What the two axes of the roofline mean and what the ridge point represents
How to compute the ridge from peak FLOPs and peak bandwidth
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.