At batch 16 and 4k context with FP8 weights, which model still fits on one 80 GB H100?

The question is a capacity puzzle: will this model and its working memory fit on one 80 GB H100? Solve it in two steps. First, weigh the **model itself**. At FP8 each parameter is roughly one byte, so a 70B model takes about 70 GB of HBM just to be loaded. A 100B model would take about 100 GB, which already exceeds the 80 GB on a single H100, before you load a single token of KV cache. That alone disqualifies the 100B option. Second, weigh the **KV cache**. For every token in every active sequence, the model stores a key and a value tensor per attention layer. The total scales as `batch * context * num_layers * 2 * d_head * H_kv * bytes_per_element`. **GQA** (Grouped-Query Attention) is the key trick: instead of having one K and V head per query head, it groups them so `H_kv = H_q / G`. With group size 8, GQA cuts the KV cache by 8x relative to standard multi-head attention. That is the reason a 70B GQA model leaves enough room in the residual ~10 GB to hold KV for 16 sequences of 4k tokens. The MoE option sounds clever but is wrong. In production serving you cannot predict which expert any future token will route to, so the full expert table must already be resident in HBM. MoE saves compute per token, it does not save memory. So the correct pick is the 70B GQA model, the only option whose weights and KV cache both fit inside the 80 GB budget.

Capacity planning on a single H100 is weights-first, then KV. **Weights at FP8.** One byte per parameter is the working approximation. 70B fits in ~70 GB. 100B does not fit at all; you would need a second GPU or aggressive INT4 quantization to make it work. **KV budget.** The residual ~10 GB on the H100 must hold the KV cache for 16 active sequences of 4k tokens, plus activation scratch and CUDA workspace. The KV formula is: $$\text{KV bytes} = 2 \cdot B \cdot L \cdot N_\text{layers} \cdot H_\text{kv} \cdot d_\text{head} \cdot \text{bytes/elem}$$ For a 70B model with **GQA group size 8** (typical for Llama-3 family), `H_kv = H_q / 8`. That single change drops the KV cache by 8x compared to vanilla MHA at the same head dim, which is precisely what lets the math close. The MoE distractor fails on a serving misconception. At decode time the router is content-dependent and per-token; you cannot pre-stage 'only the active experts' because any token can route to any expert. Production MoE deployments (DeepSeek, Mixtral, GPT-OSS, Llama-4-Maverick) hold the full expert table in HBM, sometimes spread across GPUs with expert parallelism. MoE buys compute amortization, not memory savings. The last distractor pretends KV cost is negligible. It is not. At long context the KV cache routinely exceeds the weight footprint, which is exactly why MLA (DeepSeek), KV quantization, and PagedAttention exist.