- 1Router scores each token and selects top-k experts per token
- 2Weighted sum of expert outputs is added into the residual stream
- 3Each expert GPU runs local FFN compute on its assigned token batch
- 4All to all dispatch sends tokens to the GPUs owning their selected experts
- 5All to all combine gathers weighted expert outputs back to origin ranks
Expert-parallel MoE forward pass: route locally → all-to-all dispatch → expert FFN compute → all-to-all combine → weighted residual add.
Picture a food court with kitchens on different floors. First you decide which counter each order needs (router). Then orders ride the elevator to the right floor (dispatch), and chefs cook there (expert compute). The results ride back (combine), and the plates merge into the meal (weighted sum into residual).
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Expert parallelism is what turns MoE from a parameter-count trick into a distributed systems problem. Tokens must travel to weights, not the reverse. The reverse sounds plausible if you think of MoE as 'just more FFN matmuls.'
The five steps (route, dispatch, compute, combine, residual add) are the canonical forward pass skeleton used in DeepSpeed-MoE, Megatron, Tutel, and serving runtimes. Getting the order right is necessary for explaining latency, stragglers, and why load balancing matters at the systems layer. Reordering steps is not a stylistic mistake. It describes an impossible execution plan.
Step 1: Router scores and top-k selection
Every token's hidden vector x projects to router logits via W_router. Softmax or sigmoid produces expert scores. Top-k selects which experts will run. Gate weights for the selected experts are stored for the final weighted sum.
This step executes on the token's origin rank, wherever the attention output currently lives in the parallel layout. No expert weights are consulted yet beyond the router's small projection matrix.
Routing must complete before dispatch because you cannot ship a token without knowing its destination expert ranks.
Router compute is cheap relative to expert FFNs: a matmul from d_model to N experts. Still, it must finish before dispatch because destination ranks are unknown until top-k returns. Pipelining sometimes overlaps router on rank i with dispatch from rank j, but the logical dependency remains.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Megatron-DeepSpeed and Tutel-style MoE training pipelines implement all-to-all dispatch before grouped expert GEMM.
- vLLM and SGLang MoE serving paths dispatch tokens to expert shards each decode step before local FFN execution.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy two all to all collectives instead of one?
Dispatch redistributes inputs by expert ownership. Combine redistributes outputs back to token origin ranks. The communication patterns differ.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Placing expert FFN compute before all-to-all dispatch. Tokens must reach the GPU that owns the expert weights first.
60 second bullets to scan on the way to the call.
Router before any cross-GPU traffic
All-to-all dispatch to expert ranks
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.