What does arithmetic intensity actually measure and what does AI near 1 imply?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Your answer

**Arithmetic intensity** (AI) is a ratio: how many arithmetic operations (FLOPs) a kernel performs per byte it reads from memory. $$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved}}$$ The number tells you which side of the system is the bottleneck. **High AI** means the kernel does lots of math on each byte it fetches. Compute is the binding constraint; the memory bus has headroom. The kernel is **compute-bound**. **Low AI** means the kernel does very little math per byte. The compute units sit idle waiting for the next chunk of data; memory bandwidth is the binding constraint. The kernel is **bandwidth-bound**. LLM decode is the canonical low-AI workload. Each forward pass reads the full set of model weights (gigabytes) and does one matmul against a single activation vector. The work per byte is close to one FLOP. That puts decode at AI near 1, which lands it deep in the bandwidth-bound region. The practical takeaway: when AI is near 1, adding more compute does nothing because the GPU is already starved waiting for memory. The fixes have to raise AI or cut bytes moved. Batching raises AI by reusing each weight read across many requests. KV cache quantization cuts bytes per token. FlashAttention reorganizes the attention computation to cut memory traffic. All three target the AI ratio, not the raw FLOPs ceiling.

**Arithmetic intensity** is the ratio of FLOPs performed to bytes moved from memory. It is the single number that decides whether a kernel is compute-bound or bandwidth-bound on a given GPU. $$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved from memory}}$$ The **roofline model** plots this against achievable performance. A GPU has two ceilings: a peak memory bandwidth ceiling and a peak compute ceiling. Below the ridge point intensity, the kernel is bounded by the bandwidth roof and scaling intensity raises throughput linearly. Above the ridge point, the kernel is bounded by the compute roof and more intensity buys nothing extra. For modern Hopper and Blackwell GPUs the ridge-point intensity sits in the hundreds of FLOPs per byte: peak FP8 compute divided by peak HBM bandwidth. Workloads with AI well below the ridge are bandwidth-bound; workloads above are compute-bound. **LLM decode sits at AI near 1.** A single decode step reads the full weight matrices (and the KV cache) from HBM, then does one matmul against a single new activation vector per request. That is one FLOP per byte at the matrix-vector level. With the ridge point in the hundreds, decode is two to three orders of magnitude below it, which is why decode latency is bound by HBM bandwidth rather than tensor-core throughput. The optimization implications follow directly. Adding FLOPs (faster tensor cores) does nothing for decode; the GPU is already starved waiting for memory. The wins come from raising AI (batching reuses each weight read across many requests, lifting AI linearly with batch size) or cutting bytes moved (FP8 KV cache, GQA, MLA, attention quantization). These are not different optimizations; they are different attacks on the same AI ratio.

Quick reference

TL;DR

Arithmetic intensity is FLOPs done per byte read from memory. High AI means compute-bound, low AI means bandwidth-bound. LLM decode sits near AI of 1, which is why it is HBM-bandwidth-limited.

Memory aid

Easy to grasp

Picture a worker hauling boxes from a warehouse to a workbench. Arithmetic intensity is how much actual work the worker does per box. If the worker hauls one box and does ten minutes of careful assembly on it, the bottleneck is the worker's hands, the warehouse runs idle. If the worker grabs a box, glances at it for one second, drops it, and runs back for another, the bottleneck is the path to the warehouse, the workbench sits empty most of the time. LLM decode is the second case. Each new token requires hauling the entire model out of HBM but only doing a tiny bit of math with it before fetching it all again for the next token. The path to the warehouse, HBM bandwidth, is the binding constraint.

Key concepts

Arithmetic intensity Roofline model Memory bandwidth

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

2 min: AI definition + roofline ridge point + LLM decode at AI near 1 + which optimizations raise AI versus cut bytes + diagnostic discipline.

Real-world examples

Real products, models, and research that use this idea.

The Berkeley roofline model (Williams et al.) is the canonical framework for reasoning about arithmetic intensity and is the basis for almost every inference-optimization deck.
vLLM's continuous batching is an explicit AI-raising move: each request added to the batch reuses the same weight read, scaling AI linearly with batch.
FlashAttention (Dao et al.) is the canonical example of a kernel that drops attention memory traffic by an order of magnitude, raising AI enough to make long-context attention practical.
DeepSeek V4's Multi-head Latent Attention compresses KV cache bytes per token to raise effective decode AI on long contexts.
NVIDIA Nsight and PyTorch Profiler both report achieved FLOPs and achieved bandwidth, letting engineers compute AI per kernel and locate it on the roofline.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the ridge point shift when you move from BF16 to FP8 on the same GPU?

QWhy is prefill compute-bound while decode is bandwidth-bound on the same model?

QHow does FlashAttention raise arithmetic intensity for attention?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing more FLOPs always means more throughput. When AI is near 1, the kernel is bandwidth-bound and adding compute does nothing. The fix is to raise AI, not to add tensor cores.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.