Does MoE reduce GPU memory footprint at inference compared to a dense model with similar active compute? Explain.
MoE saves per-token FLOPs via top-k routing but typically loads all N expert weights into GPU memory, sparsity is a compute trick, not a RAM trick.
A MoE model is like a toolbox with eight specialist wrenches, but you only use two per job. You still need to carry the whole toolbox to the work site, you cannot leave six wrenches at home and hope the right one magically appears when needed. Sparsity means less work per turn of the bolt, not a smaller toolbox.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The MoE inference memory myth, "sparsity means less GPU RAM", is among the most persistent misconceptions in LLM systems interviews. Candidates hear that only top-k experts activate and infer that only top-k expert weights load. That conflates conditional compute with conditional storage.
Production MoE serving is a systems problem. The router must evaluate all experts every token. Any token in the batch might need any expert on the next step. Standard inference engines therefore keep the full expert bank resident in GPU memory, sharded across devices.
This deep dive explains what sparsity actually saves, why memory tracks total parameters, and how to provision inference correctly.
The inference memory myth survives because marketing highlights sparse activation without separating FLOPs from HBM. This short-answer variant asks you to state the correction cleanly: all N experts resident, k compute per token. The deep dive gives you the systems story to defend that sentence against skeptical interviewers.
Pair this short-answer with the spot-error sibling mentally, same distinction (FLOPs vs HBM), different format. Master once, answer both.
Closing sentence to memorize: provision GPUs on total stored parameters; estimate per-token latency from active FLOPs at k, never swap those two.
What top-k routing actually controls
Conditional compute. In each MoE layer, the router projects the token hidden state to N expert logits, selects top-k, and runs only those k expert FFN matmuls. The remaining N−k experts skip compute for that token. Per-token FLOPs in the FFN sublayer scale with k, not N.
This is the core MoE value proposition: decouple total model capacity (many specialists) from per-token work (few specialists active). A model with 8 experts of 7B-class FFN size can behave like a ~13B active model while storing ~47B total parameters.
What routing does not control is which weight tensors must be present in memory. The weights for all N experts exist in the checkpoint and must be accessible for routing decisions across the full input distribution.
Router must score all N experts every token, that requires all weight tensors addressable in HBM or fast unified memory. You cannot score expert 17 without expert 17's weights reachable at microsecond scale.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral 8×7B inference requires GPUs sized for ~47B total params, not ~13B active.
- DeepSeek-V2 serving guides provision memory for the full expert bank across multi-GPU shards.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen might expert offloading to CPU ever make sense?
Very low QPS, memory-constrained edge devices, or research prototypes, not production LLM serving at scale.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming only top-k expert weights are loaded per token, standard serving keeps the full expert bank resident.
60 second bullets to scan on the way to the call.
Sparsity saves FLOPs not RAM
All experts typically resident at inference
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.