The roofline model in plain language, without the chart, what is it saying?

Flashcard·Easy·4.0 · 0·~30s·Asked atJane Street Qdrant Sourcegraph·Relevant atNVIDIA

Attempt it

The roofline model plots achievable throughput (FLOP/s) on the y-axis against arithmetic intensity (FLOPs per byte moved from memory) on the x-axis. Two ceilings cap performance: a flat peak-compute roof at the top, and a sloped peak-bandwidth line on the left where throughput equals bandwidth times intensity. Where a kernel sits under the lower of the two ceilings tells you whether to optimize for FLOPs (compute-bound) or for memory traffic (bandwidth-bound).

Your answer

The **roofline model** is a performance picture that shows whether your kernel is limited by compute or by memory bandwidth. It is a 2D plot. The x-axis is **arithmetic intensity**: how many FLOPs the kernel does per byte it moves from memory (HBM on a GPU). The y-axis is **achievable throughput** in FLOP/s. Two ceilings cap performance. The **peak-compute roof** is a flat line at the GPU's max FLOP/s (for H100, around 1000 TFLOPS BF16). The **peak-bandwidth slope** is a diagonal line where FLOP/s equals bandwidth times intensity (for H100, around 3 TB/s). A kernel sits under the lower of the two ceilings. Far left (low intensity), the slope is lower, so you are **bandwidth-bound**. Far right (high intensity), the roof is lower, so you are **compute-bound**. The crossing point is the **knee**. For LLM inference, **prefill** has high intensity and is compute-bound. **Decode** has low intensity and is bandwidth-bound. Most inference optimization choices follow from this split.

The **roofline model** is a one-picture explanation for why a kernel runs as fast as it does (or does not). **Axes.** X-axis is arithmetic intensity, FLOPs per byte moved from off-chip memory. Y-axis is achievable throughput in FLOP/s. Both axes are usually log-scale because intensity spans several orders of magnitude across realistic workloads. **Two ceilings.** A flat horizontal line at the top is the peak-compute ceiling (the hardware's max FLOP/s). A diagonal line rising from the origin at slope equal to peak bandwidth is the peak-bandwidth ceiling. The two meet at the **knee**, the arithmetic-intensity value where compute and bandwidth balance. **Interpretation.** Left of the knee: bandwidth-bound. Right of the knee: compute-bound. The best possible performance is the lower of the two ceilings at that intensity. A real kernel typically reaches 50-90 percent of it, depending on kernel quality. **In LLM inference.** Prefill on long prompts has high intensity (one weight read per layer reused across thousands of input tokens) and sits far right of the knee on H100 and B200, so it is compute-bound. Decode reads the same weights and KV cache per token but works on single-token activations, so it sits far left of the knee, firmly bandwidth-bound. This split drives optimization choices. Lower-precision compute (FP8 on H100, FP4 on B200) raises the compute ceiling and helps prefill. KV-cache reduction (GQA, MLA) lifts decode's intensity by reducing bytes moved. Batching raises decode's effective intensity by sharing weight reads across requests.

TL;DR

Roofline plots achievable FLOP/s against arithmetic intensity. Two ceilings (peak compute, peak bandwidth) cap performance. Where a kernel sits shows compute-bound or bandwidth-bound.

Memory aid

Easy to grasp

Imagine a factory has two limits. The first is how many workers it has (compute). The second is how fast trucks can deliver raw materials (bandwidth). If your product needs only a few materials but tons of assembly work, workers are the bottleneck. If it needs tons of materials but barely any assembly, the trucks are the bottleneck. The roofline is a picture that puts both limits on one chart and lets you see at a glance which one is slowing your factory. For LLM inference, prefill is worker-bound (compute), and decode is truck-bound (bandwidth).

Key concepts

Roofline model Arithmetic intensity Prefill Decode Memory bandwidth

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The roofline model is one of the most useful performance frameworks an LLM systems engineer can use. Every optimization decision in inference (quantization, KV-cache compression, batching, speculative decoding, tensor parallelism) can be traced back to its effect on a kernel's position relative to the roofline. Engineers who reason about performance without it often choose the wrong optimizations.

Prefill and decode sit on opposite sides of the knee, which is why they need different optimizations. Memory-hierarchy effects such as FlashAttention's SRAM tiling add more ceilings to the same picture.

Arithmetic intensity and the two ceilings

Define arithmetic intensity I as the ratio of useful FLOPs a kernel performs to the bytes it moves from off-chip memory (HBM on a GPU). It is a property of the kernel and the data access pattern, measured in FLOPs per byte.

A GPU has two hard limits on what it can do.

Peak compute P_c, measured in FLOP/s. For H100 BF16 this is around 1000 TFLOPS and for FP8 about 2 PFLOPS. For B200 it is about 9 to 10 PFLOPS dense NVFP4.
Peak bandwidth P_b, measured in bytes/s. For H100 HBM3 this is roughly 3 TB/s. For B200 HBM3e, it is 8 TB/s.

Achievable performance for a kernel with intensity I is bounded by

\text{Perf}(I) = \min(P_c, P_b \cdot I)

The knee is where the two terms balance: I_knee = P_c / P_b. For H100 BF16, I_knee is around 330 FLOPs per byte. Kernels with intensity above the knee are limited by peak compute. Kernels below the knee are limited by peak bandwidth.

Why two ceilings

Whichever resource (FLOPs or bandwidth) runs out first causes the bottleneck. The roofline model captures both in one picture so you can see at a glance which one limits the kernel and how much headroom remains.

Applying the roofline to LLM inference

Optimization levers by regime, with examples

Where the simple roofline breaks down, and how the framework extends

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

H100 has roughly 1000 TFLOPS BF16 peak compute and roughly 3 TB/s HBM3 bandwidth, putting the knee around 330 FLOPs per byte.
B200 raises both numbers (about 2.25 PFLOPS dense BF16 per GPU in HGX B200, 8 TB/s HBM3e), so its BF16 knee is about 281 FLOPs per byte. Dense FP8 doubles compute and NVFP4 doubles it again.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does FlashAttention's SRAM tiling change a kernel's position on the roofline?

Naive attention is HBM-bandwidth-bound because each softmax read/write hits HBM. FlashAttention tiles QKV blocks into SRAM, reusing them across the softmax pass. Effective intensity rises because bytes from HBM drop sharply. The kernel moves closer to or onto the compute roof. The achieved win on H100 is 2-4x on long sequences.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing high FLOP/s with good performance. Hitting the bandwidth ceiling means you cannot go faster regardless of how many FLOPs your kernel does.

60 second bullets to scan on the way to the call.

Define arithmetic intensity.
Name the two roofline ceilings and what each represents.

Primary sources. Browse if you want the original framing.