What does arithmetic intensity actually measure and what does AI near 1 imply?

Flashcard·Easy·4.0 · 0·~30s·Asked atKpmg Lepton Ai Typeface·Relevant atNVIDIA

Attempt it

Your answer

**Arithmetic intensity** (AI) is a ratio: how many arithmetic operations (FLOPs) a kernel performs per byte it reads from memory. $$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved}}$$ The number tells you which side of the system is the bottleneck. **High AI** means the kernel does lots of math on each byte it fetches. Compute is the binding constraint; the memory bus has headroom. The kernel is **compute-bound**. **Low AI** means the kernel does very little math per byte. The compute units sit idle waiting for the next chunk of data; memory bandwidth is the binding constraint. The kernel is **bandwidth-bound**. LLM decode is the canonical low-AI workload. Each forward pass reads the full set of model weights (gigabytes) and does one matmul against a single activation vector. The work per byte is close to one FLOP. That puts decode at AI near 1, which lands it deep in the bandwidth-bound region. The practical takeaway: when AI is near 1, adding more compute does nothing because the GPU is already starved waiting for memory. The fixes have to raise AI or cut bytes moved. Batching raises AI by reusing each weight read across many requests. KV cache quantization cuts bytes per token. FlashAttention reorganizes the attention computation to cut memory traffic. All three target the AI ratio, not the raw FLOPs ceiling.

**Arithmetic intensity** is the ratio of FLOPs performed to bytes moved from memory. It is the single number that decides whether a kernel is compute-bound or bandwidth-bound on a given GPU. $$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved from memory}}$$ The **roofline model** plots this against achievable performance. A GPU has two ceilings: a peak memory bandwidth ceiling and a peak compute ceiling. Below the ridge point intensity, the kernel is bounded by the bandwidth roof and scaling intensity raises throughput linearly. Above the ridge point, the kernel is bounded by the compute roof and more intensity buys nothing extra. For modern Hopper and Blackwell GPUs the ridge-point intensity sits in the hundreds of FLOPs per byte: peak FP8 compute divided by peak HBM bandwidth. Workloads with AI well below the ridge are bandwidth-bound; workloads above are compute-bound. **LLM decode sits at AI near 1.** A single decode step reads the full weight matrices (and the KV cache) from HBM, then does one matmul against a single new activation vector per request. That is one FLOP per byte at the matrix-vector level. With the ridge point in the hundreds, decode is two to three orders of magnitude below it, which is why decode latency is bound by HBM bandwidth rather than tensor-core throughput. The optimization implications follow directly. Adding FLOPs (faster tensor cores) does nothing for decode; the GPU is already starved waiting for memory. The wins come from raising AI (batching reuses each weight read across many requests, lifting AI linearly with batch size) or cutting bytes moved (FP8 KV cache, GQA, MLA, attention quantization). These are not different optimizations; they are different attacks on the same AI ratio.

TL;DR

Arithmetic intensity is FLOPs done per byte read from memory. High AI means compute-bound, low AI means bandwidth-bound. LLM decode sits near AI of 1, which is why it is HBM-bandwidth-limited.

Memory aid

Easy to grasp

Picture a worker hauling boxes from a warehouse to a workbench. Arithmetic intensity is how much actual work the worker does per box. If the worker hauls one box and does ten minutes of careful assembly on it, the bottleneck is the worker's hands, the warehouse runs idle. If the worker grabs a box, glances at it for one second, drops it, and runs back for another, the bottleneck is the path to the warehouse, the workbench sits empty most of the time. LLM decode is the second case. Each new token requires hauling the entire model out of HBM but only doing a tiny bit of math with it before fetching it all again for the next token. The path to the warehouse, HBM bandwidth, is the binding constraint.

Key concepts

Arithmetic intensity Roofline model Memory bandwidth

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Arithmetic intensity is the single most important number in inference performance reasoning. It is not a microoptimization detail; it is the framework that decides which optimizations apply to which workloads. A serving engineer who reasons in AI knows why batching matters for decode and not for prefill, why FlashAttention dropped attention from quadratic to linear memory, why FP8 KV cache compounds with W8A8 quantization. A serving engineer who does not reason in AI keeps reaching for faster tensor cores when the bottleneck is HBM bandwidth.

The definition is simple: FLOPs per byte moved from memory. The implications are deep. The roofline model turns AI into a visual map of where every kernel sits between two ceilings, and every inference optimization can be located as either raising the AI of a workload or moving the ceilings.

This deep dive walks the definition, the roofline picture, where common workloads land on it, and how the optimization landscape maps onto AI-raising versus byte-cutting moves. By the end you should be able to estimate AI for a kernel, place it on the roofline, and pick the right optimization for whichever side of the ridge it lives on.

The definition and the units

Arithmetic intensity is a ratio with explicit units:

\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved from memory}}

The numerator counts arithmetic operations: adds and multiplies that the compute units execute. The denominator counts bytes that move from a slower level of the memory hierarchy to a faster one, typically HBM to SRAM on a GPU, though the same framework applies to L2-to-register or DDR-to-cache on other architectures.

Units matter. AI is often reported as FLOPs per byte, with no further qualification, but the byte count depends on which memory level you draw the line at. HBM-to-SRAM is the dominant cost on modern GPUs, so most LLM inference literature uses that boundary. A kernel that hits in L2 cache moves bytes from L2 to SRAM at much higher bandwidth, so its effective AI looks different. Pick the boundary that matches your bottleneck and report it explicitly.

The number itself is intuition-friendly. AI of 1 means one FLOP per byte: read a byte, do one operation, move on. AI of 100 means a hundred operations per byte: pull a byte in, work it hard, then read the next one. The higher the number, the more arithmetic happens for each byte the memory system has to deliver.

The roofline and the ridge point

Where LLM workloads sit

Mapping optimizations onto AI

Diagnostic discipline

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

The Berkeley roofline model (Williams et al.) is the canonical framework for reasoning about arithmetic intensity and is the basis for almost every inference-optimization deck.
vLLM's continuous batching is an explicit AI-raising move: each request added to the batch reuses the same weight read, scaling AI linearly with batch.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the ridge point shift when you move from BF16 to FP8 on the same GPU?

FP8 doubles peak compute throughput at the same bandwidth, so the ridge point intensity doubles. A workload that was just at the ridge on BF16 becomes bandwidth-bound on FP8 unless it also cuts bytes moved (FP8 KV cache, FP8 activations).

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing more FLOPs always means more throughput. When AI is near 1, the kernel is bandwidth-bound and adding compute does nothing. The fix is to raise AI, not to add tensor cores.

60 second bullets to scan on the way to the call.

The definition of arithmetic intensity as FLOPs divided by bytes moved
How the ridge point of a GPU is computed from peak FLOPs and peak bandwidth

Primary sources. Browse if you want the original framing.