Memory Bandwidth

Speculative decoding is already on and batch size just climbed to 64, keep it or kill it?

Walk through how tensor parallelism shards one matmul across GPUs and what it adds to decode step time

What unit measures GPU memory bandwidth and why does that number cap decode speed?

What does arithmetic intensity actually measure and what does AI near 1 imply?

Pair HBM and SRAM with their roles on an inference GPU

Decode-phase attention is memory bandwidth bound. Explain what flips the regime.

Why does prefill saturate compute while decode is bottlenecked on memory bandwidth?

Where does FlashAttention deliver the biggest serving speedup?

In LLM serving, what is the primary driver of end to end latency for a generation request?