Pair HBM and SRAM with their roles on an inference GPU
Drag each answer to line up with its matching prompt
HBM
Model weights and the running KV cache
SRAM
HBM read bandwidth
Lives in HBM during inference
Per block tiles used inside fused kernels like FlashAttention
Lives in SRAM during inference
Static RAM, small on die scratchpad per SM
Bandwidth ceiling that decides decode TPOT
High Bandwidth Memory, stacked off die DRAM
HBM is the big off-die DRAM holding weights and KV cache; SRAM is the tiny on-die scratchpad where fused kernels keep tiles, and HBM bandwidth is what caps decode TPOT.
Picture a busy chef in a tiny kitchen next to a huge warehouse. The warehouse has every ingredient on the planet, but it is across the parking lot and takes time to walk to. The kitchen counter only fits a few pans, but it is right next to the stove. A smart chef plans meals so most of the chopping and mixing happens on the counter, and trips to the warehouse stay rare. A GPU is the same setup. The warehouse is HBM, the counter is SRAM, and the chef is a kernel. Decoding feels slow when every token forces another warehouse run, and clever attention algorithms keep more work on the counter to make it feel fast.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
HBM and SRAM are the two memory tiers behind almost every LLM serving decision. Every optimization in the inference stack, from FlashAttention to paged KV cache to quantization, is a strategy for managing the gap between these two tiers. Without a clear mental model of which data lives where and which bandwidth caps which phase, it is hard to explain why batch size matters, why quantization helps, or why decode differs from prefill.
SRAM is small but sits next to the math units, so reads from it are fast. HBM holds everything the model needs, but each read from it takes much longer. A fast kernel does as much work as possible on data already in SRAM and reads from HBM as few times as possible.
What the acronyms mean physically
HBM expands to High Bandwidth Memory. It is a stack of DRAM dies mounted next to the GPU die on a silicon interposer, connected through extremely wide buses. On H100, you get 80 GB of HBM3 at roughly 3.3 TB/s. B200 with HBM3e raises both numbers. The wide and stacked layout is what lets DRAM hit terabyte-per-second bandwidth that ordinary DDR memory cannot approach.
SRAM expands to Static RAM. It is on-die memory built into each streaming multiprocessor, accessed at speeds comparable to the math units themselves. CUDA programmers call it shared memory. Capacity per SM is in the low hundreds of kilobytes (228 KB per SM on H100), and a chip has on the order of 100 SMs, so total SRAM is roughly tens of megabytes.
The size and speed asymmetry is what makes the hierarchy matter. HBM is large (80 GB) but slow per byte. SRAM is fast but tiny per SM. There is also L2 cache between the two tiers (around 50 MB on H100), which the hardware manages automatically but kernel authors can hint at.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- FlashAttention by Tri Dao tiles attention into SRAM on H100 and B200, avoiding the HBM round trips that naive attention pays.
- vLLM's PagedAttention keeps the KV cache organized in HBM blocks so concurrent Llama 3.1 70B requests pack densely.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is FlashAttention faster than naive attention even though it does the same math?
Naive attention materializes the full N-by-N score matrix in HBM, costing O(N^2) memory reads and writes. FlashAttention tiles Q, K, and V into SRAM, computes a running softmax block by block, and only writes the final output back. Same FLOPs, far fewer HBM round trips.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating HBM and SRAM as interchangeable cache tiers. HBM is the bulk store at gigabytes; SRAM is per-SM and measured in hundreds of kilobytes.
60 second bullets to scan on the way to the call.
Acronym expansions for HBM and SRAM
Approximate capacities and bandwidths on H100 and B200
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.