Batching

Walk through how CUDA graphs reshape per-step decode latency and when the win disappears

In benchmarks, tokens per second measures what exactly?

Throughput, for an LLM serving system: give the working definition

What does a padding mask zero out, and why is it needed when batching variable-length sequences?

You pad a batch to equal length. How does the model know to ignore the filler tokens?

Spot the errors in this claim about speculative decoding

Apply the roofline model to LLM inference and identify where decode and prefill sit on it.

Why are output tokens typically 3-5× more expensive than input tokens?

Spot the errors in this 'optimize decode by reducing FLOPs' proposal

Walk through the byte accounting that proves a single-batch decode step is bandwidth-bound on H100.

Why did continuous batching replace static batching for LLM serving?

Predict the critical batch size where decode crosses from memory-bound to compute-bound on H100

Why is batching the single biggest throughput lever for LLM decode?

What is the primary reason batching is so much more effective for LLM decode than for CNN inference?

As batch size grows, where does throughput stop increasing and per-request latency start exploding?

Predict the throughput and per-request latency behaviour as batch size grows past saturation

Spot the errors in this 'batching is universal' claim

Why does batching help LLM decode disproportionately more than batching helps CNN inference?

What is the arithmetic intensity of an LLM decode step and why is it close to 1?

Predict the arithmetic intensity of decode at varying batch sizes