- 1Router scores each token and selects top-k experts per token
- 2Weighted sum of expert outputs is added into the residual stream
- 3Each expert GPU runs local FFN compute on its assigned token batch
- 4All to all dispatch sends tokens to the GPUs owning their selected experts
- 5All to all combine gathers weighted expert outputs back to origin ranks
Expert-parallel MoE forward pass: route locally → all to all dispatch → expert FFN compute → all to all combine → weighted residual add.
Like a food court with kitchens on different floors: first you decide which counter each order needs (router), then orders ride the elevator to the right floor (dispatch), chefs cook there (expert compute), results ride back (combine), and plates merge into the meal (weighted sum into residual).
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Expert parallelism is what turns MoE from a parameter-count trick into a distributed systems problem. The order question tests whether you understand that tokens must travel to weights, not the reverse, a mistake that sounds plausible if you think of MoE as 'just more FFN matmuls.'
The five steps in this question, route, dispatch, compute, combine, residual add, are the canonical forward pass skeleton used in DeepSpeed-MoE, Megatron, Tutel, and serving runtimes. Getting the order right is necessary for explaining latency, stragglers, and why load balancing matters at the systems layer.
This deep dive walks each step, names the collectives, and connects to production bottlenecks.
Order questions expose whether you have traced a MoE forward pass on real hardware or only studied diagrams. The five-step sequence, route, dispatch, compute, combine, residual, is how DeepSpeed-MoE, Megatron-Core, and serving runtimes schedule work. Reordering steps is not a stylistic mistake; it describes an impossible execution plan.
Walk through the five steps aloud before interviews, muscle memory prevents swapping combine and residual under pressure.
Step 0: Router scores and top-k selection
Every token's hidden vector x projects to router logits via W_router. Softmax or sigmoid produces expert scores; top-k selects which experts will run. Gate weights for the selected experts are stored for the final weighted sum.
This step executes on the token's origin rank, wherever the attention output currently lives in the parallel layout. No expert weights are consulted yet beyond the router's small projection matrix.
Routing must complete before dispatch because you cannot ship a token without knowing its destination expert ranks.
Router compute is cheap relative to expert FFNs, a matmul from d_model to N experts. Still, it must finish before dispatch because destination ranks are unknown until top-k returns. Pipelining sometimes overlaps router on rank i with dispatch from rank j, but logical dependency remains.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Megatron-DeepSpeed and Tutel-style MoE training pipelines implement all to all dispatch before grouped expert GEMM.
- vLLM and SGLang MoE serving paths dispatch tokens to expert shards each decode step before local FFN execution.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy two all to all collectives instead of one?
Dispatch redistributes inputs by expert ownership; combine redistributes outputs back to token origin ranks, different communication patterns.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Placing expert FFN compute before all to all dispatch, tokens must reach the GPU that owns the expert weights first.
60 second bullets to scan on the way to the call.
Router before any cross-GPU traffic
All to all dispatch to expert ranks
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.