Calculate the per token KV cache footprint of a Llama-3-8B style config in fp16
Your serving team sizes GPU memory for a Llama-3-8B style model: 32 layers, 8 KV heads (GQA), head_dim 128, cache held in fp16 (2 bytes per value). Predict the KV cache bytes per token, and the total cache for one 8192-token sequence.
Per token: 2 x 32 x 8 x 128 x 2 = 131072 bytes = 128 KB. Per 8192-token sequence: 128 KB x 8192 = 1 GiB.
Each token in the conversation needs to keep a pair of small notebooks at every layer of the model: one for keys, one for values. At each layer, each notebook is 8 KV heads wide and 128 numbers deep. Two notebooks, 32 layers, 8 heads, 128 numbers each, two bytes per number in fp16. Multiply it out: 131072 bytes per token, which is 128 KB. An 8000-token conversation is 8000 of these notebooks, which sums to 1 gibibyte. Now imagine running 32 such conversations at once; that is 32 GB of pure cache, on top of the model weights. This arithmetic, and the GQA trick that shrinks the KV-heads term from 32 to 8, is exactly why modern open LLMs can serve long contexts on single GPUs.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This calculation is the single most-asked LLM serving arithmetic question because it forces you to combine architecture knowledge (Llama 3 uses GQA with 8 KV heads), bookkeeping (the factor of 2 for K and V together), and unit conversion (bytes to KB to GiB). Get it right and the rest of LLM serving capacity planning falls out of the same arithmetic.
The answer is 131072 bytes per token (128 KB) and 1 GiB per 8192-token sequence. This walkthrough derives those numbers, sanity-checks them against the powers of 2, compares against the MHA and fp8-cache alternatives, and ends with the H100 capacity implications.
Step through the arithmetic
The formula:
Plug in Llama 3 8B values: L=32, H_kv=8 (GQA), d_head=128, b=2 (fp16).
Compute by grouping powers of 2. Notice that all four numerical factors are themselves powers of 2: 2 = 2^1, 32 = 2^5, 8 = 2^3, 128 = 2^7, 2 = 2^1. Sum the exponents: 1 + 5 + 3 + 7 + 1 = 17. So the per-token cache is 2^17 = 131072 bytes.
Convert to KB: 131072 / 1024 = 128 KB. This is the canonical Llama 3 8B per-token cache number you will see quoted in vLLM and TRT-LLM documentation.
For 8192 tokens: 131072 x 8192 = 2^17 x 2^13 = 2^30 = 1073741824 bytes = 1 GiB exactly. The fact that 8192 is a power of 2 makes this come out as a round binary number.
A quick sanity check: if you ever get a non-power-of-2 cache total for a power-of-2 context length, something is off. The Llama 3 8B numbers happen to align perfectly with binary units, which is convenient mnemonically.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 8B on a single H100 (80 GB): cache budget around 56 GB after weights and overhead, allowing about 50 concurrent 8k-context sequences in fp16.
- vLLM ships fp8 cache for Hopper deployments; the 1 GiB per 8k sequence drops to 512 MiB, doubling effective concurrency.
What an interviewer would ask next. Try answering before peeking at the approach.
QAt what batch size does the KV cache equal the weight footprint of Llama 3 8B at 8192 tokens?
Weights are 16 GB in fp16. Cache per 8k sequence is 1 GiB. Cache hits 16 GB at batch 16. Beyond that, cache dominates total memory. This is the knee where serving optimisation shifts from 'fit the weights' to 'maximise cache utilisation'.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Dropping the leading 2 (giving 64 KB instead of 128 KB), using 32 KV heads instead of 8 (overshooting 4x), or confusing 1 GiB (binary) with 1 GB (decimal); under bf16 the cache is 1073741824 bytes, which is 1 GiB exactly.
60 second bullets to scan on the way to the call.
The formula bytes_per_token = 2 x L x H_kv x d_head x b and each term's role
The specific Llama 3 8B numbers: L=32, H_kv=8, d_head=128
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.