Why is quantizing MoE models harder than quantizing dense models of the same active compute size?
MoE quantization is harder because N expert banks need separate calibration, router precision is routing-critical, and wrong expert selection compounds quality loss.
Quantizing a dense model is like compressing one big textbook. Quantizing MoE is like compressing eight specialist textbooks plus a librarian who decides which book to open. If the librarian's decision gets fuzzy, you open the wrong book entirely. That is a much bigger mistake than a small typo on the right page.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Quantization is already a sensitivity analysis problem for dense LLMs. MoE multiplies that problem by N expert banks and adds a routing decision gate whose precision determines which bank executes. Quantization becomes a per-component calibration challenge, not just a bit-width reduction.
Wrong expert selection is a discrete failure mode far more damaging than matmul rounding on the correct expert. Per-expert specialization makes activation statistics diverge. That invalidates one-scale-fits-all calibration.
MoE quantization is N calibration surfaces plus a discrete gate. Dense quantization struggles with one activation distribution. MoE splits distributions across specialists and lets the router pick the bank. A wrong top-k flip is worse than matmul noise, and that ordering drives production mixed-precision policies.
Per-expert calibration: why one scale fails
Through training, MoE experts specialize. Expert 3 may mostly serve code tokens with activation magnitudes in one range. Expert 7 may serve dialogue with a different distribution. INT4/FP8 quantization maps floating-point weights and activations to low-bit formats using per-tensor or per-channel scale factors.
A global scale optimized on the aggregate activation histogram clips outliers for specialists with wide ranges and wastes precision for specialists with narrow ranges. Per-expert calibration runs representative token subsets through each expert's expected routing path and tunes scales independently.
Dense models avoid this because one FFN weight matrix sees the full token distribution. MoE splits that distribution across N banks, each needing its own calibration surface.
Per-expert calibration workflow: route representative tokens to each expert, collect activation max/absmax, tune scales per bank. A uniform global scale clips specialists with wide ranges.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- NVIDIA TensorRT-LLM MoE quantization uses per-expert FP8 scaling with router in higher precision.
- vLLM FP8 MoE support documents separate weight scales per expert shard.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure routing stability under quantization?
Compare top-k expert indices FP16 vs quantized on a calibration set. Report flip rate per layer.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Applying one global calibration scale to all experts. Each specialist sees different activation distributions.
60 second bullets to scan on the way to the call.
Per-expert activation distribution differences
Separate calibration per expert bank
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.