A modern inference GPU has two main memory tiers, and matching the right data to the right tier is most of the performance story.
**HBM** stands for High Bandwidth Memory. It is the large stack of DRAM mounted next to the GPU die through silicon interposers. An H100 has 80 GB of HBM3, a B200 has more. This is where model weights and the running KV cache live during inference. HBM has enormous bandwidth by DRAM standards, around 3 to 8 TB/s on modern accelerators, but it is still far slower than the math units can consume.
**SRAM** stands for Static RAM. It is the tiny on-die scratchpad attached to each streaming multiprocessor, often referred to as shared memory in CUDA. Capacity is measured in hundreds of kilobytes per SM, not gigabytes. The win is latency: SRAM sits right next to the math units, so reads and writes are essentially free compared with an HBM trip.
Fused kernels like FlashAttention exist because they tile attention into SRAM and stream scores there, avoiding the round trip to HBM that the naive attention algorithm pays. The HBM bandwidth ceiling is what caps decode tokens per second.
**HBM** and **SRAM** are the two memory tiers that matter for GPU inference, and the right mental model is bulk and fast versus tiny and fast.
HBM, or High Bandwidth Memory, is the off-die DRAM stacked next to the GPU chip. An H100 has 80 GB at about 3.3 TB/s. A B200 increases both. This is where model weights and the entire KV cache live. It is the big store, the only memory large enough to hold a multi-billion-parameter model.
SRAM is the on-die static RAM serving each streaming multiprocessor. Capacity is in the low hundreds of kilobytes per SM, but access latency is single-digit cycles. It is the kernel's scratchpad, used for tiles, intermediate accumulators, and shared values between threads in a block.
Decode is HBM-bandwidth-bound. Every step reads the model weights and the KV cache from HBM to generate one token per sequence. That stream rate is the ceiling on tokens per second per request. Compute units sit idle waiting on memory, which is why batching and quantization, both of which reduce bytes per token, are the highest-leverage knobs.
FlashAttention's contribution is precisely about respecting the hierarchy. It tiles Q, K, and V into SRAM, computes the softmax incrementally, and never materializes the full attention matrix in HBM.
Quick reference
TL;DR
HBM is the big off-die DRAM holding weights and KV cache; SRAM is the tiny on-die scratchpad where fused kernels keep tiles, and HBM bandwidth is what caps decode TPOT.
Picture a busy chef in a tiny kitchen next to a huge warehouse. The warehouse has every ingredient on the planet, but it is across the parking lot and takes time to walk to. The kitchen counter only fits a few pans, but it is right next to the stove. A smart chef plans meals so most of the chopping and mixing happens on the counter, and trips to the warehouse stay rare. A GPU is the same setup. The warehouse is HBM, the counter is SRAM, and the chef is a kernel. Decoding feels slow when every token forces another warehouse run, and clever attention algorithms keep more work on the counter to make it feel fast.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Asked about CPU inference or non-NVIDIA hardware The HBM and SRAM terms are GPU-specific. CPU stacks use DRAM and L1/L2/L3 caches with different tradeoffs.
Diagnosing a prefill-bound workload Prefill can be compute-bound, so HBM bandwidth is not the dominant ceiling; tensor-core utilization matters more.
Reasoning about training memory Training adds gradients, optimizer state, and activation memory that change the HBM accounting; the decode-time framing does not transfer cleanly.
Quick scan · 2–4 min
Everything important, quickly.
3 min: HBM as off-die DRAM holding weights and KV + SRAM as per-SM scratchpad + decode is HBM-bandwidth-bound + FlashAttention tiles into SRAM + why batching and quantization help.
Real-world examples
Real products, models, and research that use this idea.
1FlashAttention by Tri Dao tiles attention into SRAM on H100 and B200, avoiding the HBM round trips that naive attention pays.
2vLLM's PagedAttention keeps the KV cache organized in HBM blocks so concurrent Llama 3.1 70B requests pack densely.
3DeepSeek V4 inference on H100 clusters hits the HBM bandwidth ceiling during decode, which is why weight quantization to FP8 helps.
4NVIDIA's B200 raises HBM3e bandwidth versus H100, directly lifting decode tokens-per-second for Claude Opus 4.7 and GPT-5.5 class models.
5TensorRT-LLM uses SRAM-resident epilogue fusion to overlap activations with weight loads, squeezing more useful work out of each HBM read.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is FlashAttention faster than naive attention even though it does the same math?
Sign in or go Pro to see the approach for this follow-up.
QWhy is decode bandwidth-bound while prefill can be compute-bound?
Sign in or go Pro to see the approach for this follow-up.
QHow does NVLink fit into this hierarchy across multiple GPUs?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating HBM and SRAM as interchangeable cache tiers. HBM is the bulk store at gigabytes; SRAM is per-SM and measured in hundreds of kilobytes.