Arithmetic Intensity

The roofline model in plain language, without the chart, what is it saying?

Walk through what actually happens during the prefill phase of an LLM forward pass.

What unit measures GPU memory bandwidth and why does that number cap decode speed?

Why do hosted LLM APIs charge separate per-million rates for input and output tokens?

FLOPs vs FLOP/s: fill in the counts versus rate distinction.

What does arithmetic intensity actually measure and what does AI near 1 imply?

Apply the roofline model to LLM inference and identify where decode and prefill sit on it.

On the roofline, which intervention moves an LLM decode workload most directly toward higher peak throughput?

Why does prefill saturate compute while decode is bottlenecked on memory bandwidth?

Which statement best explains why prefill and decode hit different bottlenecks on the same GPU?

Why does prefill's O(seq^2) attention cost saturate compute rather than bandwidth?

Why is HBM bandwidth the binding constraint for LLM decode and how do you reason about the 'memory wall'?

Why are output tokens typically 3-5× more expensive than input tokens?

Spot the errors in this 'optimize decode by reducing FLOPs' proposal

Why is 'reduce FLOPs to speed up decode' the most common beginner misconception in LLM serving?

Walk through the byte accounting that proves a single-batch decode step is bandwidth-bound on H100.

Predict the bandwidth-vs-compute latency of a single Llama-70B decode step on H100

Predict the critical batch size where decode crosses from memory-bound to compute-bound on H100

Why is batching the single biggest throughput lever for LLM decode?

What is the primary reason batching is so much more effective for LLM decode than for CNN inference?

As batch size grows, where does throughput stop increasing and per-request latency start exploding?

Spot the errors in this 'batching is universal' claim

Why does batching help LLM decode disproportionately more than batching helps CNN inference?

What is the arithmetic intensity of an LLM decode step and why is it close to 1?

Predict the arithmetic intensity of decode at varying batch sizes