Pick the cheaper to serve option at equal quality: Llama 4 Maverick MoE or Llama 3.1 70B dense

Attempt it

At a fixed quality target where both models score within a few points on your eval, choose between Llama 4 Maverick (MoE: large total params, small active-params subset) and Llama 3.1 70B dense. State the conditions under which each wins on tokens per dollar at decode, and name the two failure modes that erase the MoE advantage.

Free · 2 AI evals / day

**Llama 4 Maverick** is a mixture of experts model with roughly 400B total parameters but only 17B active per token. **Llama 3.1 70B** is a dense model where every parameter participates in every token. The right cost lens for decode is active parameters, not total. Maverick reads 17B parameters of expert weights per token plus the shared attention weights. Llama 3.1 70B reads 70B parameters per token. On bandwidth-bound decode, Maverick moves fewer bytes per token, so it can be faster and cheaper. The catch is that the full 400B set still has to live in HBM, because the router could activate any expert on any token. On H100 (80 GB HBM), Maverick in FP8 needs about 400 GB total, so you cannot fit it on one or two GPUs. You need a multi-GPU setup (4-8 H100s with expert parallel) or an H200 (141 GB) or B200 (192 GB) where the model fits more compactly. Maverick wins on tokens per dollar when batch is moderate to large and HBM holds the full expert set. It loses at batch 1-2 (router overhead dominates) and when experts spill to CPU or NVMe.

**The cost lens.** Decode cost scales with active parameters, not total. Maverick activates roughly 17B parameters per token (a few experts plus shared attention). Llama 3.1 70B activates all 70B. On bandwidth-bound decode, this is a 4x advantage in bytes-moved per token for Maverick. **Where Maverick wins.** Moderate-to-large batch serving on hardware where the full 400B expert set fits in HBM. The sweet spot is H200 or B200 single-node or a tight H100 cluster with expert parallel. At batch 16-64, the router-dispatch overhead is amortized across many requests and the active-params advantage cleanly translates to tokens per dollar. **Failure mode 1: small batch.** At batch 1 or 2, the MoE router has to dispatch each token's experts independently. The fixed cost of routing, top-k expert selection, and all to all communication for expert parallel dominates the saved compute. A dense 70B model has none of this overhead and runs cleaner GEMMs end to end. Llama 3.1 70B beats Maverick on per-stream latency at low concurrency. **Failure mode 2: HBM spill.** Maverick's full 400B parameters at FP8 occupy roughly 400 GB. On 2x H100 (160 GB combined) the model does not fit. The runtime either splits experts across many GPUs (adding inter-GPU traffic) or pages experts from CPU memory. CPU paging collapses decode latency by 10-100x because PCIe is two orders of magnitude slower than HBM. Llama 3.1 70B fits in 70 GB FP8 and runs cleanly on a single H100. **Practical rule.** High-throughput serving on H200 or B200 with steady traffic and batch 16+: pick Maverick. Low-traffic, latency-sensitive deployment on H100: pick Llama 3.1 70B. The MoE advantage is conditional on both batch and HBM. Confirm both before committing.

Quick reference

TL;DR

MoE wins on tokens per dollar when batch is moderate to large and full expert set fits in HBM. It loses at batch 1-2 from router overhead and on H100 from expert spill.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture two restaurants serving the same menu. The dense restaurant has a small kitchen where every dish uses the same set of ingredients on the same prep counter. The MoE restaurant has a huge pantry with many specialty stations, but each order only uses two or three stations. The MoE saves work per dish because most stations stay idle for any given order. That savings is real when the pantry fits in the kitchen and there are lots of orders to amortize the runners who carry ingredients to the right station. If the pantry has to live in a back warehouse, the runners take forever and the savings disappear. If only one order comes in at a time, the runner overhead is bigger than the saved prep work.

Key concepts

Mixture of expertsLlama 4 Active parameters

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: anchor active versus total parameters as the cost lens, walk through Maverick's 17B-active versus 70B dense, name the two failure modes (small-batch router overhead and HBM spill), and close with a hardware-specific decision rule (H200/B200 for Maverick, H100 for dense).

Real-world examples

Real products, models, and research that use this idea.

Meta's Llama 4 Maverick (400B total, 17B active) targets H200 and B200 deployments where the full expert set fits in single-node HBM.
DeepSeek V4 uses similar MoE math, with serving stacks tuned for batch 32-128 on H800 clusters.
Mistral Large 3 ships as a dense model precisely because it targets latency-sensitive single-stream use cases on H100.
Together AI's hosted Maverick endpoint runs on 8x H200 nodes with expert parallel and prices per active-param equivalent.
OpenAI's GPT-5.5 architecture is widely believed to use MoE for similar bandwidth-economic reasons at frontier scale.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does expert parallelism distribute the 400B weights across GPUs without breaking decode bandwidth?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the role of load-balance loss in MoE training, and how does it affect inference?

Sign in or go Pro to see the approach for this follow-up.

QWhy does Maverick at FP8 not gain a clean 4x speedup over dense 70B in practice?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Comparing total parameter counts. Maverick has 400B total but only 17B active per token, so decode cost compares to a 17B-class dense model, not a 400B one.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.