Sketch the roofline model and place LLM prefill and LLM decode (at varying batch sizes) on it. Use H100 numbers (peak ~3 TB/s HBM, ~989 TFLOPs FP16). Identify where each workload's performance is bound and how interventions move them on the plot.
The roofline plots attainable FLOP/s against arithmetic intensity. Decode sits far left on the bandwidth slope, prefill sits higher, and batching shoves decode rightward toward the compute roof.
Imagine a highway carrying parts from a warehouse to a factory. The factory assembles incredibly fast, but only if parts keep arriving. If each truck carries one tiny part, the factory sits idle waiting for deliveries, so the road is the bottleneck. If each truck is packed full, the factory finally runs flat out and the assembly line becomes the limit. The roofline is a chart of this tradeoff. The slanted part says you are limited by the road, the flat ceiling says you are limited by the factory. LLM decode is the under-loaded truck reading huge weights to make one token. Batching packs more work onto each delivery, pushing you off the road limit toward the factory ceiling.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The roofline model is the single most useful mental tool for reasoning about LLM serving performance, because it collapses a messy hardware question into one chart with one decision: is this kernel limited by compute or by memory bandwidth? Almost every serving optimization that matters in 2026 is, at bottom, a move on the roofline plot.
The model has two axes. The horizontal axis is arithmetic intensity, the number of floating-point operations performed per byte fetched from off-chip memory. The vertical axis is attainable throughput, measured in FLOP/s. A workload is a single point on this plane, and the chart's two roofs tell you the best you could possibly do at that point.
This deep dive derives the ridge point on an H100, places LLM prefill and decode on the plot, scans decode across batch sizes to watch it slide from the bandwidth slope toward the compute roof, and then catalogs how quantization, grouped-query attention, FlashAttention, and speculative decoding each move a workload. By the end you should be able to draw the plot from memory, derive the critical intensity, and explain in one sentence what the model tells you to optimize.
The two roofs and the attainable-throughput formula
The roofline says attainable throughput is the lower of two ceilings. One ceiling is the chip's peak compute, a flat horizontal line. The other is the memory-bandwidth limit, a line rising from the origin whose slope is the bandwidth. At low intensity the sloped line is lower, so you are bandwidth-bound. At high intensity the flat line is lower, so you are compute-bound.
The compact statement is:
The intuition is direct. If a kernel does only one FLOP per byte, then no matter how fast the tensor cores are, you can only feed them as fast as HBM delivers bytes. The sloped roof is the bandwidth speaking.
If a kernel does thousands of FLOPs per byte, bandwidth stops mattering and you saturate the compute units, hitting the flat roof. The whole game in inference is knowing which roof you are under, because that dictates which hardware resource is worth buying and which software knob is worth turning.
One subtlety is worth stating early. Arithmetic intensity is a property of the workload, not the chip. The same matmul can be bandwidth-bound on one batch shape and compute-bound on another. The roofs belong to the hardware; the point belongs to the kernel. Optimization is the art of dragging that point toward whichever roof currently limits it, or of lowering the roof itself so the same point attains more.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM uses continuous batching to keep decode arithmetic intensity high under real traffic, pushing the workload rightward toward the H100 compute roof.
- DeepSeek V4 ships Multi-head Latent Attention to shrink KV bytes streamed each decode step, lifting the effective bandwidth roof on the roofline.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does decode arithmetic intensity scale with batch size but prefill intensity does not?
Decode reads all weights once per step regardless of batch, then does work proportional to batch, so intensity rises with B. Prefill already amortizes weight reads across the sequence length, so its intensity is set by sequence length, not request batching.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating decode as compute-bound and optimizing FLOPs. Decode at batch 1 hits well under one percent of peak compute; it is starved by HBM bandwidth, so the fix is raising arithmetic intensity, not adding more math units.
60 second bullets to scan on the way to the call.
The attainable throughput formula and what each roof represents
How to compute the ridge point from peak compute and bandwidth
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.