Roofline Model

TPOT is high, so the team plans to upgrade A100 to a faster compute GPU, critique

Find the wrong move in 'decode is slow, so let's switch to a smaller FLOP model'

Match Hopper FP8 and Blackwell FP4 tensor cores to their throughput and bandwidth effects on decode

What unit measures GPU memory bandwidth and why does that number cap decode speed?

What does arithmetic intensity actually measure and what does AI near 1 imply?

Match each weight/activation quant regime to its mechanism and best fit workload

Apply the roofline model to LLM inference and identify where decode and prefill sit on it.

On the roofline, which intervention moves an LLM decode workload most directly toward higher peak throughput?

Why does prefill saturate compute while decode is bottlenecked on memory bandwidth?

PremiumWhich statement best explains…

Why does prefill's O(seq^2) attention cost saturate compute rather than bandwidth?

Why is the vision encoder phase a different bottleneck class from LLM decode in a multi-modal model?

Why is HBM bandwidth the binding constraint for LLM decode and how do you reason about the 'memory wall'?

Spot the errors in this 'optimize decode by reducing FLOPs' proposal

PremiumWhy is 'reduce FLOPs…

Predict the bandwidth vs compute latency of a single Llama-70B decode step on H100

Predict the critical batch size where decode crosses from memory bound to compute bound on H100

Why is batching the single biggest throughput lever for LLM decode?

PremiumWhat is the primary…

As batch size grows, where does throughput stop increasing and per request latency start exploding?

Predict the throughput and per request latency behaviour as batch size grows past saturation

Spot the errors in this 'batching is universal' claim

Why does batching help LLM decode disproportionately more than batching helps CNN inference?

What is the arithmetic intensity of an LLM decode step and why is it close to 1?

Predict the arithmetic intensity of decode at varying batch sizes