At MoE inference, which resource constraints apply?
Correct: all N experts load into memory (A), FLOPs scale with k (B), and MoE serving bottlenecks differ from dense (D). Wrong: bandwidth irrelevant (C) and router not loaded (E).
MoE is like a library with eight specialist books on the shelf but you only read two per visit. The shelf must hold all eight books (memory), you only read two (compute), and the librarian's index card (router) is always on the desk too.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MoE inference resource accounting is a three-dimensional problem: memory (total params stored), compute (active FLOPs per token), and bandwidth (weight loading and KV cache access). The multi-select tests whether you can identify which statements are true (A, B, D) and which embed the common memory myth (C, E).
Candidates who select C reveal they believe only k experts are stored. Those who select E forget the router exists. Both errors conflate sparse activation with sparse storage.
This deep dive explains each option, builds the resource decoupling model, and shows how to profile MoE serving bottlenecks.
The memory myth, believing only k experts are stored at inference, is the single most common MoE misconception in systems interviews. It leads teams to under-provision GPU memory and then wonder why a "13B active" model needs 80GB HBM. This deep dive builds the three-dimensional resource model (memory, compute, bandwidth) and explains why A, B, and D are true while C and E embed specific errors.
Walk through this model when an interviewer asks "how do you size GPUs for MoE serving?" The answer always starts with total stored parameters and KV cache, then checks whether active FLOPs or memory bandwidth becomes the binding constraint at your target batch size and context length.
Senior interviewers reward candidates who connect this mechanism to a concrete deployment or training decision, not only the textbook definition. Close with one number, one failure mode, and one monitoring signal you would track in production.
Option A: all N experts in memory
Correct. Standard MoE inference loads the complete expert bank into GPU memory, sharded across devices. Any token in any request might route to any expert. Conditional compute (only k run) does not imply conditional storage (only k loaded).
This is the foundational MoE inference fact. It drives GPU provisioning: total parameters determine HBM requirements, not active parameters. Mixtral needs GPUs sized for ~47B stored weights even though active compute is ~13B class.
Provisioning implication. GPU sizing for MoE serving uses total stored parameters, not active parameters. A deployment team provisioning for "13B active" on Mixtral will run out of HBM because the checkpoint contains all 8 expert FFN banks plus attention, embeddings, and router weights. Expert parallelism shards the full bank across devices, but every shard must be resident, any token in any request might route to any expert on that shard.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral serving: ~47B stored (memory-bound), ~13B active (compute-efficient).
- vLLM MoE loads all expert shards per GPU regardless of per-token k.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen is MoE memory-bound vs compute-bound at inference?
Memory-bound when HBM fills with all experts + KV cache before GPU compute saturates; compute-bound when k FFN matmuls max out SM utilization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Selecting option C (bandwidth irrelevant) or E (router not loaded), both confuse sparse compute with sparse storage.
60 second bullets to scan on the way to the call.
All N experts resident in memory
FLOPs scale with k not N
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.