Memory Bandwidth Bound

TPOT is high, so the team plans to upgrade A100 to a faster compute GPU, critique

Find the wrong move in 'decode is slow, so let's switch to a smaller-FLOP model'

Apply the roofline model to LLM inference and identify where decode and prefill sit on it.

On the roofline, which intervention moves an LLM decode workload most directly toward higher peak throughput?

Why are output tokens typically 3-5× more expensive than input tokens?

Spot the errors in this 'optimize decode by reducing FLOPs' proposal

Why is 'reduce FLOPs to speed up decode' the most common beginner misconception in LLM serving?

For a 200-token prompt and 1000-token completion, why does decode dominate end-to-end latency?

Walk through the byte accounting that proves a single-batch decode step is bandwidth-bound on H100.

Predict the bandwidth-vs-compute latency of a single Llama-70B decode step on H100

Predict the critical batch size where decode crosses from memory-bound to compute-bound on H100

Spot the errors in this 'API is slow because of network and tokenizer' explanation

What is the primary reason batching is so much more effective for LLM decode than for CNN inference?

Spot the errors in this 'batching is universal' claim

Why does batching help LLM decode disproportionately more than batching helps CNN inference?

Predict the arithmetic intensity of decode at varying batch sizes