Pair HBM and SRAM with their roles on an inference GPU

A modern inference GPU has two main memory tiers, and matching the right data to the right tier is most of the performance story. **HBM** stands for High Bandwidth Memory. It is the large stack of DRAM mounted next to the GPU die through silicon interposers. An H100 has 80 GB of HBM3, a B200 has more. This is where model weights and the running KV cache live during inference. HBM has enormous bandwidth by DRAM standards, around 3 to 8 TB/s on modern accelerators, but it is still far slower than the math units can consume. **SRAM** stands for Static RAM. It is the tiny on-die scratchpad attached to each streaming multiprocessor, often referred to as shared memory in CUDA. Capacity is measured in hundreds of kilobytes per SM, not gigabytes. The win is latency: SRAM sits right next to the math units, so reads and writes are essentially free compared with an HBM trip. Fused kernels like FlashAttention exist because they tile attention into SRAM and stream scores there, avoiding the round trip to HBM that the naive attention algorithm pays. The HBM bandwidth ceiling is what caps decode tokens per second.

**HBM** and **SRAM** are the two memory tiers that matter for GPU inference, and the right mental model is bulk and fast versus tiny and fast. HBM, or High Bandwidth Memory, is the off-die DRAM stacked next to the GPU chip. An H100 has 80 GB at about 3.3 TB/s. A B200 increases both. This is where model weights and the entire KV cache live. It is the big store, the only memory large enough to hold a multi-billion-parameter model. SRAM is the on-die static RAM serving each streaming multiprocessor. Capacity is in the low hundreds of kilobytes per SM, but access latency is single-digit cycles. It is the kernel's scratchpad, used for tiles, intermediate accumulators, and shared values between threads in a block. Decode is HBM-bandwidth-bound. Every step reads the model weights and the KV cache from HBM to generate one token per sequence. That stream rate is the ceiling on tokens per second per request. Compute units sit idle waiting on memory, which is why batching and quantization, both of which reduce bytes per token, are the highest-leverage knobs. FlashAttention's contribution is precisely about respecting the hierarchy. It tiles Q, K, and V into SRAM, computes the softmax incrementally, and never materializes the full attention matrix in HBM.