Select the correct comparisons of an MoE transformer versus a dense transformer at equal active parameters

Multi-select·Hard·4.0 · 0·~1 min·Asked atPolyai Pwc Qdrant·Relevant atDatabricks Google Mistral AI Sarvam

Attempt it

Compare an MoE transformer to a dense one with the same **active parameter count** (meaning both do roughly the same amount of work per token). The differences are in what changes between them and what stays the same. **What MoE gains.** Total parameters scale with the number of experts. If a dense model has 7B params and the MoE has 8 experts in each FFN, the MoE might have ~47B total params but still run ~13B active per token. The extra params let the model store more knowledge (different experts can specialize on different patterns) without paying for it in per-token compute. **What MoE costs.** Every expert weight has to sit in GPU HBM, whether or not a given token uses it. Serving a 47B-param MoE needs roughly the same HBM as a 47B dense model, even though only 13B is active per token. That makes MoE expensive to deploy on small clusters. **The latency catch.** When the batch size is small (e.g., single-request serving), sparse routing fragments the matmul into smaller per-expert calls and GPU utilization drops. At large batch sizes the experts saturate and the cost disappears. **What does not change.** Attention is identical between dense and MoE; the block recipe (pre-norm, residual) is identical; only the FFN sublayer body changes.

**At equal active params, MoE separates total params from active FLOPs in a way dense cannot.** MoE total params scale with expert count. Mixtral 8x7B is 47B total but ~13B active. Llama 4 Maverick is ~400B total but ~17B active. A dense model at 17B active is just 17B total; no separation. **Where MoE wins.** Quality per active FLOP at scale. The extra total params give the model more capacity for storing patterns and weak expert specialization, which empirically improves quality at the same active-FLOP budget. This is why frontier labs ship MoE: Mixtral, Llama 4, DeepSeek V3, Qwen2.5-MoE all use it. **Where MoE costs.** Three places. First, HBM: every expert weight must be resident on a GPU, even unused experts. Serving cost scales with total params for memory, not active params. Second, batch-1 latency: sparse routing fragments the FFN matmul into per-expert sub-matmuls, hurting tensor-core utilization at small batches. Third, training infrastructure: load-balancing auxiliary loss, expert parallelism across GPUs, and all to all communication for token routing add real engineering complexity. **What does not change.** Attention is bit-identical between dense and MoE. The block recipe (pre-norm, residual wrap) is unchanged. Only the FFN body is swapped. Anyone claiming MoE 'replaces attention' or 'rewrites the block' is wrong. The correct framing is: same active compute, more total capacity, more total memory, worse small-batch utilization, identical attention.

TL;DR

At equal active params, MoE wins on quality-per-FLOP at scale but loses on HBM (must store all experts) and on small-batch latency (routing fragments matmul).

Memory aid

Easy to grasp

Think of dense as a well-organized worker who keeps every skill in their head; limited by how much they can memorize. MoE is more like a small office of specialists: more total knowledge stored on the shelves, but only one or two of them get pulled in for any given task, so the per-task work stays the same. The catch is that all the specialists have to stay in the office (HBM) even when they aren't working, and when only one customer walks in at a time (batch 1), the receptionist's handoff becomes a bottleneck. Llama 4 Maverick is 128 specialists with top-1 routing; vast shelf, one specialist per token.

Key concepts

Transformer architecture Mixture of Experts (MoE)

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Comparing MoE to dense at equal active parameters is the most important architectural decision when sizing a model in 2026. The right framing is not 'which is faster' or 'which is bigger' but 'which separation does the workload favor': MoE separates total params from active FLOPs in a way dense cannot, and that separation has both upsides and downsides depending on how the model is trained and served.

A serious answer covers four axes: quality per active FLOP, HBM and serving cost, batch-size sensitivity, and training infrastructure. It also flags what is not different: attention, the block recipe, and the residual stream are identical between MoE and dense variants in the same family.

The parameter separation: total vs active

Dense models have one parameter count. The number of parameters loaded into HBM equals the number used per token equals the FLOP budget. There is no separation.

MoE breaks this. Total params scale with the number of experts; active params scale with k (the number of experts run per token). The relationship is:

\text{total\_params} = \text{attention} + \text{embeddings} + N \cdot \text{params\_per\_expert\_ffn}

\text{active\_params\_per\_token} = \text{attention} + \text{embeddings} + k \cdot \text{params\_per\_expert\_ffn}

The ratio total / active is the sparsity multiplier. For Mixtral 8x7B: total 47B / active 13B ≈ 3.6x. For Llama 4 Maverick: total 400B / active 17B ≈ 24x. For DeepSeek V3: total 671B / active 37B ≈ 18x.

Why this matters. Active FLOPs determine compute cost; total params determine memory cost. Dense models tie these together. MoE lets you increase capacity (total) without increasing per-token compute (active), at the price of paying for the capacity in HBM.

Where MoE wins: quality per active FLOP at scale

Where MoE pays: HBM, batch-1 latency, and training cost

What does not change between MoE and dense

When to pick which

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Dimension	Dense (at X active params)	MoE (at X active params)
Total parameters	X	X · (N / k), much larger
Active FLOPs per token	X-equivalent	~X-equivalent (similar)
HBM footprint	Scales with X	Scales with total, not active
Quality on benchmarks	Baseline	Matches 2-3x larger dense
Batch-1 latency	Good (clean matmul)	Worse (fragmented per-expert)
Batch-128 latency	Linear in batch	Often competitive once experts saturate
Training infrastructure	Standard DP + TP	Adds expert parallelism + all to all

Real products, models, and research that use this idea.

Llama 4 Maverick (128 experts, top-1) has ~400B total / ~17B active; needs ~6 H200 GPUs for fp16 serving despite being 'only' 17B active.
Mixtral 8x7B (8 experts, top-2) has ~47B total / ~13B active; fits on 2 H100 80GB with fp16 but needs more than a Mistral 7B dense equivalent would.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf MoE total params scale with N, can you keep adding experts indefinitely?

No. Beyond a point, the router cannot meaningfully differentiate among experts and load-balancing fails. Empirically, returns flatten around 64-128 experts; DeepSeek goes further with fine-grained + shared expert designs.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Conflating total params with active params, or thinking MoE 'replaces the FFN' means it replaces attention too. The swap is FFN-only and the param separation is the entire point of the architecture.

60 second bullets to scan on the way to the call.

The total vs active parameter separation that MoE introduces
Why HBM cost scales with total params, not active params

Primary sources. Browse if you want the original framing.