Why does prefill's O(seq^2) attention cost saturate compute rather than bandwidth?

A GPU has two limits: how fast it can do math, in FLOPs per second, and how fast it can read data from memory, in bytes per second. For any operation, whichever limit you hit first decides your speed. The deciding ratio is arithmetic intensity, the FLOPs done per byte read. **Prefill** is the phase where the model reads your whole prompt at once. Because all the prompt tokens go through together, the model can read each weight matrix from memory one time and then apply it to every token. One expensive memory read serves many tokens of math. That reuse is the whole story. When bytes get reused many times, the math units stay busy and the slow memory reads get hidden. So even though attention's cost grows like the square of the prompt length, that cost is in math operations, not in extra memory traffic. Decode, generating one token at a time, has the opposite shape. Each step reads huge weights and the whole cache just to produce a single token, with almost no reuse. There the memory reads dominate, so decode is bandwidth-bound while prefill is compute-bound.

The trap is reading O(seq^2) as a memory cost. It is a FLOP count. Arithmetic intensity, not raw cost, decides the bottleneck regime. **Prefill** runs the whole prompt through one parallel forward pass. For the projection and feed-forward matmuls, each weight matrix is read from memory once and applied to all seq tokens. Intensity therefore scales with seq, roughly 2000 for a 2k prompt. That is far right of the roofline ridge, so these matmuls saturate the FLOP roof. The attention matmul looks scarier but behaves the same way. The score step and the value step are large batched matmuls. Each K row and each V row is reused across many Q rows. With FlashAttention the seq by seq score matrix never touches HBM. It is tiled in SRAM, so the quadratic term costs FLOPs, not bytes. Decode is the mirror image. Each step processes one token, reads the full weights and cache, and reuses nothing. Intensity is near one, which is bandwidth-bound. The takeaway is that cost scaling and intensity are different axes. Prefill reuses its bytes heavily, so it lives on the compute side.