Pair HBM and SRAM with their roles on an inference GPU
Drag each answer to line up with its matching prompt
HBM
Model weights and the running KV cache
SRAM
HBM read bandwidth
Lives in HBM during inference
Per block tiles used inside fused kernels like FlashAttention
Lives in SRAM during inference
Static RAM, small on die scratchpad per SM
Bandwidth ceiling that decides decode TPOT
High Bandwidth Memory, stacked off die DRAM
HBM is the big off-die DRAM holding weights and KV cache; SRAM is the tiny on-die scratchpad where fused kernels keep tiles, and HBM bandwidth is what caps decode TPOT.
Picture a busy chef in a tiny kitchen next to a huge warehouse. The warehouse has every ingredient on the planet, but it is across the parking lot and takes time to walk to. The kitchen counter only fits a few pans, but it is right next to the stove. A smart chef plans meals so most of the chopping and mixing happens on the counter, and trips to the warehouse stay rare. A GPU is the same setup. The warehouse is HBM, the counter is SRAM, and the chef is a kernel. Decoding feels slow when every token forces another warehouse run, and clever attention algorithms keep more work on the counter to make it feel fast.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
HBM and SRAM are the two acronyms you cannot skip in a serving-engineering interview. Every other optimization in the LLM inference stack, from FlashAttention to paged KV cache to quantization, is a strategy for managing the gap between these two memory tiers. Without a clear mental model of which data lives where and which bandwidth caps which phase, you cannot explain why batch size matters, why quantization helps, or why decode is fundamentally different from prefill.
The right starting picture is a busy chef in a tiny kitchen next to a huge warehouse. The kitchen counter is SRAM. It is small, but the chef can grab anything on it instantly. The warehouse is HBM. It holds every ingredient the chef could need, but each trip costs time. The chef's job is to plan recipes so that as much work as possible happens on the counter and trips to the warehouse are minimized.
This deep dive walks through the physical layout, the capacities and bandwidths, what lives where during inference, why decode is HBM-bandwidth-bound, and how FlashAttention exploits the SRAM tier to dodge the bottleneck.
What the acronyms mean physically
HBM expands to High Bandwidth Memory. It is a stack of DRAM dies mounted next to the GPU die on a silicon interposer, connected through extremely wide buses. On H100, you get 80 GB of HBM3 at roughly 3.3 TB/s. B200 with HBM3e pushes both numbers higher. The wide and stacked layout is what lets DRAM hit terabyte-per-second bandwidth that ordinary DDR memory cannot approach.
SRAM expands to Static RAM. It is on-die memory built into each streaming multiprocessor, accessed at speeds comparable to the math units themselves. CUDA programmers call it shared memory. Capacity per SM is in the low hundreds of kilobytes (228 KB per SM on H100), and a chip has on the order of 100 SMs, so total SRAM is roughly tens of megabytes.
The size and speed asymmetry is what makes the hierarchy matter. HBM is large (80 GB) but slow per byte. SRAM is fast (essentially free relative to math) but tiny per SM. There is also L2 cache between the two tiers (around 50 MB on H100), which the hardware manages automatically but kernel authors can hint at.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- FlashAttention by Tri Dao tiles attention into SRAM on H100 and B200, avoiding the HBM round trips that naive attention pays.
- vLLM's PagedAttention keeps the KV cache organized in HBM blocks so concurrent Llama 3.1 70B requests pack densely.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is FlashAttention faster than naive attention even though it does the same math?
Naive attention materializes the full N-by-N score matrix in HBM, costing O(N^2) memory reads and writes. FlashAttention tiles Q, K, and V into SRAM, computes a running softmax block by block, and only writes the final output back. Same FLOPs, far fewer HBM round trips.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating HBM and SRAM as interchangeable cache tiers. HBM is the bulk store at gigabytes; SRAM is per-SM and measured in hundreds of kilobytes.
60 second bullets to scan on the way to the call.
Acronym expansions for HBM and SRAM
Approximate capacities and bandwidths on H100 and B200
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.