The roofline model in plain language, without the chart, what is it saying?

Flashcard·Easy·4.0 · 0·~30s·Asked atJane Street Qdrant Sourcegraph·Relevant atNVIDIA

Attempt it

The roofline model plots achievable throughput (FLOP/s) on the y-axis against arithmetic intensity (FLOPs per byte moved from memory) on the x-axis. Two ceilings cap performance: a flat peak-compute roof at the top, and a sloped peak-bandwidth line on the left where throughput equals bandwidth times intensity. Where a kernel sits under the lower of the two ceilings tells you whether to optimize for FLOPs (compute-bound) or for memory traffic (bandwidth-bound).

Your answer

The **roofline model** is a performance picture that tells you, at a glance, whether your kernel is limited by compute or by memory bandwidth. It is a 2D plot. The x-axis is **arithmetic intensity**: how many FLOPs the kernel does per byte it moves from memory (HBM on a GPU). The y-axis is **achievable throughput**, measured in FLOP/s. Two ceilings cap what you can possibly hit. One is the **peak-compute roof**, a horizontal line at the top showing the GPU's max FLOP/s (for H100, around 1000 TFLOPS BF16). The other is the **peak-bandwidth slope**, a diagonal line on the left where achievable FLOP/s equals bandwidth times arithmetic intensity (for H100, around 3 TB/s HBM bandwidth). Any real kernel sits under the lower of the two ceilings at its arithmetic-intensity value. If you are far left on the x-axis (low intensity), the bandwidth slope is below the compute roof, so you are **bandwidth-bound**. If you are far right (high intensity), the compute roof is below the slope, so you are **compute-bound**. The transition point is called the **knee** of the roofline. For LLM inference, **prefill** has high intensity (lots of FLOPs per byte) and is compute-bound. **Decode** has low intensity (few FLOPs per byte) and is bandwidth-bound. The roofline framing is the foundation for every optimization decision in this space.

The **roofline model** is a one-picture explanation for why a kernel runs as fast as it does (or does not). **Axes.** X-axis is arithmetic intensity, FLOPs per byte moved from off-chip memory. Y-axis is achievable throughput in FLOP/s. Both axes are usually log-scale because intensity spans several orders of magnitude across realistic workloads. **Two ceilings.** A flat horizontal line at the top is the peak-compute ceiling (the hardware's max FLOP/s). A diagonal line rising from the origin at slope equal to peak bandwidth is the peak-bandwidth ceiling. The two meet at the **knee**, the arithmetic-intensity value where compute and bandwidth balance perfectly. **Interpretation.** Where a kernel's intensity lands relative to the knee tells you the regime. Left of the knee: bandwidth-bound. Right of the knee: compute-bound. The actual achieved performance is the minimum of the two ceilings at that intensity value, and a real kernel typically hits 50-90 percent of that ceiling depending on quality. **Why it matters for LLM inference.** Prefill on long prompts has high intensity (one weight read per layer reused across thousands of input tokens) and sits far right of the knee on H100 / B200, so it is compute-bound. Decode reads the same weights and KV cache per token but operates on single-token activations, so intensity is low and it sits left of the knee, firmly bandwidth-bound. This split drives every optimization decision: quantization (FP8 on H100, FP4 on B200) shifts the compute ceiling and helps prefill; KV-cache reduction (GQA, MLA, paged attention) lifts decode's intensity by reducing bytes moved; batching raises decode's effective intensity by amortizing weight reads across requests.

TL;DR

Roofline plots achievable FLOP/s against arithmetic intensity. Two ceilings (peak compute, peak bandwidth) cap performance; where a kernel sits says compute-bound or bandwidth-bound.

Memory aid

Easy to grasp

Imagine a factory has two limits. The first is how many workers it has (compute). The second is how fast trucks can deliver raw materials (bandwidth). If your product needs only a few materials but tons of assembly work, workers are the bottleneck. If it needs tons of materials but barely any assembly, the trucks are the bottleneck. The roofline is a picture that puts both limits on one chart and lets you see at a glance which one is choking your factory. For LLM inference, prefill is worker-bound (compute), and decode is truck-bound (bandwidth).

Key concepts

Roofline model Arithmetic intensity Prefill Decode Memory bandwidth

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The roofline model is the single most important performance framework an LLM systems engineer can carry. Every optimization decision in inference (quantization, KV-cache compression, batching, speculative decoding, tensor parallelism) can be traced back to its effect on a kernel's position relative to the roofline. Engineers who reason about performance without it tend to chase the wrong levers.

This deep dive walks through the model from first principles, applies it to the prefill versus decode split that defines LLM serving, and shows how the framework extends to cover modern memory-hierarchy effects like FlashAttention's SRAM tiling.

Arithmetic intensity and the two ceilings

Define arithmetic intensity I as the ratio of useful FLOPs a kernel performs to the bytes it moves from off-chip memory (HBM on a GPU). It is a property of the kernel and the data access pattern, measured in FLOPs per byte.

A GPU has two hard limits on what it can do.

Peak compute P_c, measured in FLOP/s. For H100 BF16 this is around 1000 TFLOPS; for FP8 about 2 PFLOPS; for B200 FP4 about 4 PFLOPS or higher.
Peak bandwidth P_b, measured in bytes/s. For H100 HBM3 this is roughly 3 TB/s; for B200 HBM3e roughly 8 TB/s.

Achievable performance for a kernel with intensity I is bounded by

\text{Perf}(I) = \min(P_c, P_b \cdot I)

The knee is where the two terms balance: I_knee = P_c / P_b. For H100 BF16, I_knee is around 330 FLOPs per byte. Kernels with intensity above the knee are limited by peak compute; below the knee, by peak bandwidth.

Why this is the right framing

Performance bottlenecks are caused by whichever resource (FLOPs or bandwidth) runs out first. The roofline model captures both in one picture so you can see at a glance which is biting and how much headroom remains.

Applying the roofline to LLM inference

Optimization levers by regime, with 2026 examples

Where the simple roofline breaks down, and how the framework extends

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

H100 has roughly 1000 TFLOPS BF16 peak compute and roughly 3 TB/s HBM3 bandwidth, putting the knee around 330 FLOPs per byte.
B200 raises both numbers (around 2.2 PFLOPS FP8, around 8 TB/s HBM3e) with the knee staying in a similar range; FP4 doubles compute again.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does FlashAttention's SRAM tiling change a kernel's position on the roofline?

Naive attention is HBM-bandwidth-bound because each softmax read/write hits HBM. FlashAttention tiles QKV blocks into SRAM, reusing them across the softmax pass. Effective intensity rises because bytes-from-HBM drops dramatically; the kernel moves closer to or onto the compute roof. The achieved win on H100 is 2-4x on long sequences.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Conflating high FLOP/s with good performance. Hitting the bandwidth ceiling means you cannot go faster regardless of how many FLOPs your kernel does.

60 second bullets to scan on the way to the call.

Define arithmetic intensity.
Name the two roofline ceilings and what each represents.

Primary sources. Browse if you want the original framing.