Pick the cheaper to serve option at equal quality: Llama 4 Maverick MoE or Llama 3.1 70B dense

Short answer·Medium·4.0 · 0·~3 min·Asked atCognizant Meta Voyage Ai·Relevant atDatabricks

Attempt it

At a fixed quality target where both models score within a few points on your eval, choose between Llama 4 Maverick (MoE: large total params, small active params subset) and Llama 3.1 70B dense. State the conditions under which each wins on tokens per dollar at decode, and name the two failure modes that erase the MoE advantage.

Free · 2 AI evals / day

**Llama 4 Maverick** is a mixture of experts model with roughly 400B total parameters but only 17B active per token. **Llama 3.1 70B** is a dense model where every parameter participates in every token. The right cost lens for decode is active parameters, not total. Maverick reads 17B parameters of expert weights per token plus the shared attention weights. Llama 3.1 70B reads 70B parameters per token. On bandwidth-bound decode, Maverick moves fewer bytes per token, so it can be faster and cheaper. The catch is that the full 400B set still has to live in HBM, because the router could activate any expert on any token. On H100 (80 GB HBM), Maverick in FP8 needs about 400 GB total, so you cannot fit it on one or two GPUs. You need a multi-GPU setup (4-8 H100s with expert parallel) or an H200 (141 GB) or B200 (192 GB) where the model fits more compactly. Maverick wins on tokens per dollar when batch is moderate to large and HBM holds the full expert set. It loses at batch 1-2 (router overhead dominates) and when experts spill to CPU or NVMe.

**The cost lens.** Decode cost scales with active parameters, not total. Maverick activates roughly 17B parameters per token (a few experts plus shared attention). Llama 3.1 70B activates all 70B. On bandwidth-bound decode, this is a 4x advantage in bytes-moved per token for Maverick. **Where Maverick wins.** Moderate-to-large batch serving on hardware where the full 400B expert set fits in HBM. The sweet spot is H200 or B200 single-node or a tight H100 cluster with expert parallel. At batch 16-64, the router-dispatch overhead is amortized across many requests and the active-params advantage cleanly translates to tokens per dollar. **Failure mode 1: small batch.** At batch 1 or 2, the MoE router has to dispatch each token's experts independently. The fixed cost of routing, top-k expert selection, and all to all communication for expert parallel dominates the saved compute. A dense 70B model has none of this overhead and runs cleaner GEMMs end to end. Llama 3.1 70B beats Maverick on per-stream latency at low concurrency. **Failure mode 2: HBM spill.** Maverick's full 400B parameters at FP8 occupy roughly 400 GB. On 2x H100 (160 GB combined) the model does not fit. The runtime either splits experts across many GPUs (adding inter-GPU traffic) or pages experts from CPU memory. CPU paging collapses decode latency by 10-100x because PCIe is two orders of magnitude slower than HBM. Llama 3.1 70B fits in 70 GB FP8 and runs cleanly on a single H100. **Practical rule.** High-throughput serving on H200 or B200 with steady traffic and batch 16+: pick Maverick. Low-traffic, latency-sensitive deployment on H100: pick Llama 3.1 70B. The MoE advantage is conditional on both batch and HBM. Confirm both before committing.

TL;DR

MoE wins on tokens per dollar when batch is moderate to large and full expert set fits in HBM. It loses at batch 1-2 from router overhead and on H100 from expert spill.

Memory aid

Easy to grasp

Picture two restaurants serving the same menu. The dense restaurant has a small kitchen where every dish uses the same set of ingredients on the same prep counter. The MoE restaurant has a huge pantry with many specialty stations, but each order only uses two or three stations. The MoE saves work per dish because most stations stay idle for any given order. That savings is real when the pantry fits in the kitchen and there are lots of orders to amortize the runners who carry ingredients to the right station. If the pantry has to live in a back warehouse, the runners take forever and the savings disappear. If only one order comes in at a time, the runner overhead is bigger than the saved prep work.

Key concepts

Mixture of expertsLlama 4 Active parameters

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Mixture of experts has changed the economics of LLM serving by decoupling how much the model learned from how much work it does per token. Llama 4 Maverick at 400B total parameters but 17B active per token captures the trick. The question is whether the trick pays off in production against a well-tuned dense alternative like Llama 3.1 70B.

The answer is: it depends on two things, batch size and HBM capacity. Outside the sweet spot where both align, MoE can lose to a smaller dense model. Inside the sweet spot, MoE wins by a meaningful margin on tokens per dollar.

This deep dive covers the mechanics of MoE inference, why active parameters drive decode cost while total parameters drive HBM occupancy, the two failure modes that erase the MoE advantage, and a concrete decision rule tied to hardware and workload shape. The math works out cleanly once you separate the two cost variables, but a lot of production teams conflate them.

MoE inference mechanics in one paragraph

On every token, a small router network reads the hidden state and outputs top-k expert selections (typically k=1 or k=2 for inference). The token's hidden state is sent to those experts, each expert applies its feed-forward block, and the outputs are weighted-summed back. Attention layers are shared across all experts. The trick is that only a small fraction of the model's total feed-forward parameters participate per token, so weight bandwidth per token drops sharply.

Llama 4 Maverick's specific shape: roughly 400B total parameters, 17B active per token, 56 transformer layers, 128 experts per MoE layer (k=1 active per token). At FP8 the total weight footprint is approximately 400 GB; per-token active weights are 17 GB. Llama 4 Scout sits below at roughly 109B total and 17B active, also MoE.

Llama 3.1 70B dense, by contrast, activates all 70B parameters on every token. Per-token weight bandwidth is 70 GB at FP8 across all 80 layers.

Active parameters drive compute, total drive HBM

Where Maverick wins on tokens per dollar

Failure mode 1: small-batch router overhead

Failure mode 2: HBM spill

A practical decision framework

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Meta's Llama 4 Maverick (400B total, 17B active) targets H200 and B200 deployments where the full expert set fits in single-node HBM.
DeepSeek V4 uses similar MoE math, with serving stacks tuned for batch 32-128 on H800 clusters.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does expert parallelism distribute the 400B weights across GPUs without breaking decode bandwidth?

Each GPU holds a subset of experts and tokens are routed via all to all communication. NVLink bandwidth (900 GB/s on H100, faster on B200) is fast enough that the all to all stays sub-millisecond at moderate batch. Beyond a single NVLink domain (8 GPUs), the cross-node traffic over InfiniBand becomes a bottleneck and throughput drops sharply. This is why MoE serving is typically single-node.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Comparing total parameter counts. Maverick has 400B total but only 17B active per token, so decode cost compares to a 17B-class dense model, not a 400B one.

60 second bullets to scan on the way to the call.

Distinguish active parameters from total parameters
Describe how decode cost scales with active params while HBM scales with total