Same topic, related formats. Practice these next.
On one 80 GB H100 at FP8, a 70B GQA model fits: weights consume ~70 GB and GQA shrinks the KV cache for B=16 / L=4k to fit the residual ~10 GB.
Think about packing a moving truck. The big furniture is the model weights, and the smaller boxes are the KV cache, one box per person riding along. An H100 is an 80-cubic-meter truck. A 70B FP8 model is furniture that fills about 70 cubic meters. You have 10 cubic meters left for the boxes. If the model uses GQA, the boxes are small (about an eighth the size), so 16 people's worth of luggage fits in those 10 meters. A 100B dense model is furniture that does not even fit before you start loading boxes. And a mixture of experts model is not like a truck where you only bring the experts you need today, the rules say every expert has to ride along because you do not know who will be needed when. Capacity planning is just: pack the heavy stuff first, count what is left, then check if the boxes fit.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
2 min: write the two-bucket budget (weights + KV + scratch <= 80 GB). 1 min: size FP8 weights for each option. 2 min: size KV using the formula with the architecture's H_kv. 1 min: kill the MoE and 'KV is negligible' distractors with the serving-routing and long-context arguments.
| Model + precision | Weight cost | Fits on one H100 80 GB? | Why |
|---|---|---|---|
| 70B GQA-8, FP8 weights, FP16 KV | ~70 GB | Yes (tight) | Residual ~10 GB holds KV for B=16, L=4k; KV quantization gives margin |
| 70B dense MHA, FP8 weights | ~70 GB | No at B=16, L=4k | KV without GQA is ~8x larger, blows the budget |
| 100B dense, FP8 weights | ~100 GB | No | Weights alone exceed 80 GB |
| 13B GQA, FP8 weights | ~13 GB | Yes (large headroom) | 67 GB free for KV, easily handles batch 32+ at 4k |
| 8x7B MoE, FP8 weights | ~52 GB total expert table | Yes for the model | All experts must be resident; can't stream active ones at decode |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming MoE serving lets you stream in only the active experts at decode time. Production routing is per-token, so all experts must already be resident in HBM, MoE saves compute, not memory.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.