Memory Bandwidth Bound

TPOT is high, so the team plans to upgrade A100 to a faster compute GPU, critique

Find the wrong move in 'decode is slow, so let's switch to a smaller FLOP model'

Apply the roofline model to LLM inference and identify where decode and prefill sit on it.

On the roofline, which intervention moves an LLM decode workload most directly toward higher peak throughput?

PremiumWhy are output tokens…

Spot the errors in this 'optimize decode by reducing FLOPs' proposal

PremiumWhy is 'reduce FLOPs…

PremiumFor a 200-token prompt…

Walk through the byte accounting that proves a single batch decode step is bandwidth bound on H100.

Predict the bandwidth vs compute latency of a single Llama-70B decode step on H100

Predict the critical batch size where decode crosses from memory bound to compute bound on H100

Spot the errors in this 'API is slow because of network and tokenizer' explanation

PremiumWhat is the primary…

Spot the errors in this 'batching is universal' claim

Why does batching help LLM decode disproportionately more than batching helps CNN inference?

Predict the arithmetic intensity of decode at varying batch sizes