Compare an MoE transformer to a dense one with the same **active parameter count** (meaning both do roughly the same amount of work per token). The differences are in what changes between them and what stays the same.
**What MoE gains.** Total parameters scale with the number of experts. If a dense model has 7B params and the MoE has 8 experts in each FFN, the MoE might have ~47B total params but still run ~13B active per token. The extra params let the model store more knowledge (different experts can specialize on different patterns) without paying for it in per-token compute.
**What MoE costs.** Every expert weight has to sit in GPU HBM, whether or not a given token uses it. Serving a 47B-param MoE needs roughly the same HBM as a 47B dense model, even though only 13B is active per token. That makes MoE expensive to deploy on small clusters.
**The latency catch.** When the batch size is small (e.g., single-request serving), sparse routing fragments the matmul into smaller per-expert calls and GPU utilization drops. At large batch sizes the experts saturate and the cost disappears.
**What does not change.** Attention is identical between dense and MoE; the block recipe (pre-norm, residual) is identical; only the FFN sublayer body changes.
**At equal active params, MoE separates total params from active FLOPs in a way dense cannot.**
MoE total params scale with expert count. Mixtral 8x7B is 47B total but ~13B active. Llama 4 Maverick is ~400B total but ~17B active. A dense model at 17B active is just 17B total; no separation.
**Where MoE wins.** Quality per active FLOP at scale. The extra total params give the model more capacity for storing patterns and weak expert specialization, which empirically improves quality at the same active-FLOP budget. This is why frontier labs ship MoE: Mixtral, Llama 4, DeepSeek V3, Qwen2.5-MoE all use it.
**Where MoE costs.** Three places. First, HBM: every expert weight must be resident on a GPU, even unused experts. Serving cost scales with total params for memory, not active params. Second, batch-1 latency: sparse routing fragments the FFN matmul into per-expert sub-matmuls, hurting tensor-core utilization at small batches. Third, training infrastructure: load-balancing auxiliary loss, expert parallelism across GPUs, and all to all communication for token routing add real engineering complexity.
**What does not change.** Attention is bit-identical between dense and MoE. The block recipe (pre-norm, residual wrap) is unchanged. Only the FFN body is swapped. Anyone claiming MoE 'replaces attention' or 'rewrites the block' is wrong.
The correct framing is: same active compute, more total capacity, more total memory, worse small-batch utilization, identical attention.
Quick reference
TL;DR
At equal active params, MoE wins on quality-per-FLOP at scale but loses on HBM (must store all experts) and on small-batch latency (routing fragments matmul).
Think of dense as a well-organized worker who keeps every skill in their head; limited by how much they can memorize. MoE is more like a small office of specialists: more total knowledge stored on the shelves, but only one or two of them get pulled in for any given task, so the per-task work stays the same. The catch is that all the specialists have to stay in the office (HBM) even when they aren't working, and when only one customer walks in at a time (batch 1), the receptionist's handoff becomes a bottleneck. Llama 4 Maverick is 128 specialists with top-1 routing; vast shelf, one specialist per token.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-GPU or consumer-GPU deployment MoE total params often exceed the HBM of a single consumer card even when active params would fit. Dense models scale more gracefully on small hardware.
Strict per-request latency targets at batch 1 Sparse routing fragments the matmul and hurts tensor-core utilization. Dense FFN is faster per-request when concurrent traffic is low.
Small parameter budgets (under 1B total) MoE overhead (router, load-balance, expert parallelism) doesn't pay off below ~1B total. Dense FFN is more parameter-efficient at this scale.
Quick scan · 2–4 min
Everything important, quickly.
4 min: explain the param separation, name the three costs (HBM, batch-1 latency, all to all training), confirm attention and block recipe are unchanged.
Side-by-side
Dimension
Dense (at X active params)
MoE (at X active params)
Total parameters
X
X · (N / k), much larger
Active FLOPs per token
X-equivalent
~X-equivalent (similar)
HBM footprint
Scales with X
Scales with total, not active
Quality on benchmarks
Baseline
Matches 2-3x larger dense
Batch-1 latency
Good (clean matmul)
Worse (fragmented per-expert)
Batch-128 latency
Linear in batch
Often competitive once experts saturate
Training infrastructure
Standard DP + TP
Adds expert parallelism + all to all
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick (128 experts, top-1) has ~400B total / ~17B active; needs ~6 H200 GPUs for fp16 serving despite being 'only' 17B active.
2Mixtral 8x7B (8 experts, top-2) has ~47B total / ~13B active; fits on 2 H100 80GB with fp16 but needs more than a Mistral 7B dense equivalent would.
3DeepSeek V3 (256 fine-grained experts + 1 shared, top-8) has ~671B total / ~37B active; the shared expert design stabilizes load balance vs Switch-style top-1.
4Qwen2.5-MoE A2.7B-14B has 14B total / 2.7B active; serving cost is closer to 14B dense than 2.7B dense despite the active count.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf MoE total params scale with N, can you keep adding experts indefinitely?
Sign in or go Pro to see the approach for this follow-up.
QHow does MoE interact with KV cache memory at long context?
Sign in or go Pro to see the approach for this follow-up.
QCould you use MoE on attention itself?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Conflating total params with active params, or thinking MoE 'replaces the FFN' means it replaces attention too. The swap is FFN-only and the param separation is the entire point of the architecture.