Apply the roofline model to LLM inference and identify where decode and prefill sit on it.

Attempt it

Sketch the roofline model and place LLM prefill and LLM decode (at varying batch sizes) on it. Use H100 numbers (peak ~3 TB/s HBM, ~989 TFLOPs FP16). Identify where each workload's performance is bound and how interventions move them on the plot.

Free · 2 AI evals / day

Quick reference

TL;DR

The roofline plots attainable FLOP/s against arithmetic intensity. Decode sits far left on the bandwidth slope, prefill sits higher, and batching shoves decode rightward toward the compute roof.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a highway carrying parts from a warehouse to a factory. The factory assembles incredibly fast, but only if parts keep arriving. If each truck carries one tiny part, the factory sits idle waiting for deliveries, so the road is the bottleneck. If each truck is packed full, the factory finally runs flat out and the assembly line becomes the limit. The roofline is a chart of this tradeoff. The slanted part says you are limited by the road, the flat ceiling says you are limited by the factory. LLM decode is the under-loaded truck reading huge weights to make one token. Batching packs more work onto each delivery, pushing you off the road limit toward the factory ceiling.

Key concepts

Roofline model Arithmetic intensity Memory-bandwidth bound Batching

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: roofline axes + ridge point derivation on H100 + place prefill and decode + batch scan rightward + interventions that raise AI or cut bytes.

Real-world examples

Real products, models, and research that use this idea.

vLLM uses continuous batching to keep decode arithmetic intensity high under real traffic, pushing the workload rightward toward the H100 compute roof.
DeepSeek V4 ships Multi-head Latent Attention to shrink KV bytes streamed each decode step, lifting the effective bandwidth roof on the roofline.
NVIDIA TensorRT-LLM applies FP8 weights and KV cache on H100 and B200, cutting bytes per element to raise effective arithmetic intensity.
Speculative decoding in Llama 4 serving verifies several draft tokens per weight read, multiplying useful tokens produced per byte streamed from HBM.
Anthropic and OpenAI prompt caching reuses prefilled KV state across requests, amortizing the bandwidth cost of long shared system prompts.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does decode arithmetic intensity scale with batch size but prefill intensity does not?

Sign in or go Pro to see the approach for this follow-up.

QHow does the KV cache complicate the simple two-roof picture during long-context decode?

Sign in or go Pro to see the approach for this follow-up.

QWhy might the real crossover into compute-bound happen well before the idealized batch of 330?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating decode as compute-bound and optimizing FLOPs. Decode at batch 1 hits well under one percent of peak compute; it is starved by HBM bandwidth, so the fix is raising arithmetic intensity, not adding more math units.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.