Select the correct comparisons of an MoE transformer versus a dense transformer at equal active parameters

Multi-select·Hard·4.0 · 0·~1 min·Relevant atDatabricks Google Mistral AI Sarvam

Attempt it

Compare an MoE transformer to a dense one with the same **active parameter count** (meaning both do roughly the same amount of work per token). The differences are in what changes between them and what stays the same. **What MoE gains.** Total parameters scale with the number of experts. If a dense model has 7B params and the MoE has 8 experts in each FFN, the MoE might have ~47B total params but still run ~13B active per token. The extra params let the model store more knowledge (different experts can specialize on different patterns) without paying for it in per-token compute. **What MoE costs.** Every expert weight has to sit in GPU HBM, whether or not a given token uses it. Serving a 47B-param MoE needs roughly the same HBM as a 47B dense model, even though only 13B is active per token. That makes MoE expensive to deploy on small clusters. **The latency catch.** When the batch size is small (e.g., single-request serving), sparse routing fragments the matmul into smaller per-expert calls and GPU utilization drops. At large batch sizes the experts saturate and the cost disappears. **What does not change.** Attention is identical between dense and MoE; the block recipe (pre-norm, residual) is identical; only the FFN sublayer body changes.

**At equal active params, MoE separates total params from active FLOPs in a way dense cannot.** MoE total params scale with expert count. Mixtral 8x7B is 47B total but ~13B active. Llama 4 Maverick is ~400B total but ~17B active. A dense model at 17B active is just 17B total; no separation. **Where MoE wins.** Quality per active FLOP at scale. The extra total params give the model more capacity for storing patterns and weak expert specialization, which empirically improves quality at the same active-FLOP budget. This is why frontier labs ship MoE: Mixtral, Llama 4, DeepSeek V3, Qwen2.5-MoE all use it. **Where MoE costs.** Three places. First, HBM: every expert weight must be resident on a GPU, even unused experts. Serving cost scales with total params for memory, not active params. Second, batch-1 latency: sparse routing fragments the FFN matmul into per-expert sub-matmuls, hurting tensor-core utilization at small batches. Third, training infrastructure: load-balancing auxiliary loss, expert parallelism across GPUs, and all to all communication for token routing add real engineering complexity. **What does not change.** Attention is bit-identical between dense and MoE. The block recipe (pre-norm, residual wrap) is unchanged. Only the FFN body is swapped. Anyone claiming MoE 'replaces attention' or 'rewrites the block' is wrong. The correct framing is: same active compute, more total capacity, more total memory, worse small-batch utilization, identical attention.

Quick reference

TL;DR

At equal active params, MoE wins on quality-per-FLOP at scale but loses on HBM (must store all experts) and on small-batch latency (routing fragments matmul).

Memory aid

Easy to grasp

Think of dense as a well-organized worker who keeps every skill in their head; limited by how much they can memorize. MoE is more like a small office of specialists: more total knowledge stored on the shelves, but only one or two of them get pulled in for any given task, so the per-task work stays the same. The catch is that all the specialists have to stay in the office (HBM) even when they aren't working, and when only one customer walks in at a time (batch 1), the receptionist's handoff becomes a bottleneck. Llama 4 Maverick is 128 specialists with top-1 routing; vast shelf, one specialist per token.

Key concepts

Transformer architecture Mixture of Experts (MoE)

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: explain the param separation, name the three costs (HBM, batch-1 latency, all to all training), confirm attention and block recipe are unchanged.

Side-by-side

Dimension	Dense (at X active params)	MoE (at X active params)
Total parameters	X	X · (N / k), much larger
Active FLOPs per token	X-equivalent	~X-equivalent (similar)
HBM footprint	Scales with X	Scales with total, not active
Quality on benchmarks	Baseline	Matches 2-3x larger dense
Batch-1 latency	Good (clean matmul)	Worse (fragmented per-expert)
Batch-128 latency	Linear in batch	Often competitive once experts saturate
Training infrastructure	Standard DP + TP	Adds expert parallelism + all to all

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick (128 experts, top-1) has ~400B total / ~17B active; needs ~6 H200 GPUs for fp16 serving despite being 'only' 17B active.
Mixtral 8x7B (8 experts, top-2) has ~47B total / ~13B active; fits on 2 H100 80GB with fp16 but needs more than a Mistral 7B dense equivalent would.
DeepSeek V3 (256 fine-grained experts + 1 shared, top-8) has ~671B total / ~37B active; the shared expert design stabilizes load balance vs Switch-style top-1.
Qwen2.5-MoE A2.7B-14B has 14B total / 2.7B active; serving cost is closer to 14B dense than 2.7B dense despite the active count.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf MoE total params scale with N, can you keep adding experts indefinitely?

QHow does MoE interact with KV cache memory at long context?

QCould you use MoE on attention itself?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Conflating total params with active params, or thinking MoE 'replaces the FFN' means it replaces attention too. The swap is FFN-only and the param separation is the entire point of the architecture.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.