Top 100 · Inference Optimization42 of 100

The roofline model in plain language, without the chart, what is it saying?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

The roofline model plots achievable throughput (FLOP/s) on the y-axis against arithmetic intensity (FLOPs per byte moved from memory) on the x-axis. Two ceilings cap performance: a flat peak-compute roof at the top, and a sloped peak-bandwidth line on the left where throughput equals bandwidth times intensity. Where a kernel sits under the lower of the two ceilings tells you whether to optimize for FLOPs (compute-bound) or for memory traffic (bandwidth-bound).

Your answer

The **roofline model** is a performance picture that tells you, at a glance, whether your kernel is limited by compute or by memory bandwidth. It is a 2D plot. The x-axis is **arithmetic intensity**: how many FLOPs the kernel does per byte it moves from memory (HBM on a GPU). The y-axis is **achievable throughput**, measured in FLOP/s. Two ceilings cap what you can possibly hit. One is the **peak-compute roof**, a horizontal line at the top showing the GPU's max FLOP/s (for H100, around 1000 TFLOPS BF16). The other is the **peak-bandwidth slope**, a diagonal line on the left where achievable FLOP/s equals bandwidth times arithmetic intensity (for H100, around 3 TB/s HBM bandwidth). Any real kernel sits under the lower of the two ceilings at its arithmetic-intensity value. If you are far left on the x-axis (low intensity), the bandwidth slope is below the compute roof, so you are **bandwidth-bound**. If you are far right (high intensity), the compute roof is below the slope, so you are **compute-bound**. The transition point is called the **knee** of the roofline. For LLM inference, **prefill** has high intensity (lots of FLOPs per byte) and is compute-bound. **Decode** has low intensity (few FLOPs per byte) and is bandwidth-bound. The roofline framing is the foundation for every optimization decision in this space.

The **roofline model** is a one-picture explanation for why a kernel runs as fast as it does (or does not). **Axes.** X-axis is arithmetic intensity, FLOPs per byte moved from off-chip memory. Y-axis is achievable throughput in FLOP/s. Both axes are usually log-scale because intensity spans several orders of magnitude across realistic workloads. **Two ceilings.** A flat horizontal line at the top is the peak-compute ceiling (the hardware's max FLOP/s). A diagonal line rising from the origin at slope equal to peak bandwidth is the peak-bandwidth ceiling. The two meet at the **knee**, the arithmetic-intensity value where compute and bandwidth balance perfectly. **Interpretation.** Where a kernel's intensity lands relative to the knee tells you the regime. Left of the knee: bandwidth-bound. Right of the knee: compute-bound. The actual achieved performance is the minimum of the two ceilings at that intensity value, and a real kernel typically hits 50-90 percent of that ceiling depending on quality. **Why it matters for LLM inference.** Prefill on long prompts has high intensity (one weight read per layer reused across thousands of input tokens) and sits far right of the knee on H100 / B200, so it is compute-bound. Decode reads the same weights and KV cache per token but operates on single-token activations, so intensity is low and it sits left of the knee, firmly bandwidth-bound. This split drives every optimization decision: quantization (FP8 on H100, FP4 on B200) shifts the compute ceiling and helps prefill; KV-cache reduction (GQA, MLA, paged attention) lifts decode's intensity by reducing bytes moved; batching raises decode's effective intensity by amortizing weight reads across requests.

Quick reference

TL;DR

Roofline plots achievable FLOP/s against arithmetic intensity. Two ceilings (peak compute, peak bandwidth) cap performance; where a kernel sits says compute-bound or bandwidth-bound.

Memory aid

Easy to grasp

Imagine a factory has two limits. The first is how many workers it has (compute). The second is how fast trucks can deliver raw materials (bandwidth). If your product needs only a few materials but tons of assembly work, workers are the bottleneck. If it needs tons of materials but barely any assembly, the trucks are the bottleneck. The roofline is a picture that puts both limits on one chart and lets you see at a glance which one is choking your factory. For LLM inference, prefill is worker-bound (compute), and decode is truck-bound (bandwidth).

Key concepts

Roofline model Arithmetic intensity Prefill Decode Memory bandwidth

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: arithmetic intensity definition + two ceilings + knee + LLM prefill vs decode positions + optimization levers by regime + FlashAttention tiling and cache-aware extensions.

Real-world examples

Real products, models, and research that use this idea.

H100 has roughly 1000 TFLOPS BF16 peak compute and roughly 3 TB/s HBM3 bandwidth, putting the knee around 330 FLOPs per byte.
B200 raises both numbers (around 2.2 PFLOPS FP8, around 8 TB/s HBM3e) with the knee staying in a similar range; FP4 doubles compute again.
Llama 3.1 70B decode on H100 sits firmly bandwidth-bound; vLLM benchmarks show GPU utilization around 30-50 percent during decode.
Prefill on the same model with a 2K-token prompt saturates tensor cores at near-peak FLOPs, sitting right of the knee.
FlashAttention-3 exploits SRAM tiling to effectively raise the bandwidth ceiling for attention; performance jumps even though HBM bandwidth is unchanged.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does FlashAttention's SRAM tiling change a kernel's position on the roofline?

QWhy does batching help decode more than it helps prefill, in roofline terms?

QWhat happens to the knee when you move from FP16 to FP8 quantization on H100?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Conflating high FLOP/s with good performance. Hitting the bandwidth ceiling means you cannot go faster regardless of how many FLOPs your kernel does.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.