How does dropless MoE dispatch differ from capacity factor token dropping?
How does dropless MoE dispatch differ from capacity factor token dropping during training?
Capacity-factor MoE drops overflow tokens when expert buffers fill; dropless MoE redistributes or reschedules so every routed token gets FFN compute, trading scheduling complexity for zero skips.
Capacity-factor routing is like a restaurant that seats only 10 people per chef, when an 11th order arrives, it goes out without the special sauce (token skips FFN, keeps residual). Dropless MoE is the manager who rearranges orders, opens overflow prep stations, or batches smarter so nobody misses their dish, harder to run the kitchen, but no customer leaves without the full meal.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Expert-parallel MoE training hits a physical constraint: each expert GPU can only process so many tokens per step efficiently. When routing is skewed, some experts receive more tokens than others. The systems question is what to do with the excess, drop it, or work harder to process everything.
Capacity-factor token dropping and dropless dispatch are the two canonical answers. This is a senior systems interview topic because it connects routing math to training quality, step-time predictability, and hardware utilization. Candidates who only know 'MoE has experts' fail here; candidates who can explain the overflow path pass.
This deep dive compares the mechanisms, names the quality leak in capacity drops, and explains why dropless MoE became a research and production priority at frontier scale.
Expert-parallel training is a queueing problem disguised as a neural architecture. When routers skew, some GPU ranks receive bursts of tokens while others idle. Capacity-factor drops and dropless dispatch are two overflow policies, one sacrifices sample completeness, the other sacrifices step-time predictability. Interviewers want you to name that trade without hand-waving about "efficient MoE."
Why per-expert buffers exist in expert-parallel training
In expert parallelism, experts are sharded across GPUs. After the router selects top-k experts per token, an all to all dispatch sends tokens to the ranks owning those experts. Each rank runs grouped FFN matmuls on its local token batch.
GPUs and MoE kernels are fastest with bounded, well-formed batches. If one expert receives 10× the tokens another receives, the hot rank becomes a straggler, the whole step waits. Capacity factor caps each expert's buffer at factor × (total_tokens / num_experts) to bound worst-case load.
When routing randomness or collapse pushes more tokens to an expert than the buffer allows, something must give. Capacity training chooses to drop overflow tokens rather than stall the step or allocate unbounded memory.
Picture a 64-GPU pod running a 128-expert model. All to all dispatch ships each token to k expert ranks. If expert 42 receives triple the mean load, rank 42 becomes the straggler, 63 GPUs wait at the NCCL barrier. Capacity factor caps buffer depth so rank 42 sheds excess tokens instead of stalling the global step.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Switch Transformer and GShard introduced capacity factors with token dropping as a standard training efficiency tool.
- Dropless MoE research and production stacks (including Tutel-style dispatch optimizations) target zero-drop training at trillion-token scale.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would you accept capacity-factor drops in production training?
Name throughput predictability, early-scale experiments, or when drop rate is monitored and below a tight threshold, not when quality-sensitive convergence is fragile.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming dropless MoE eliminates all routing imbalance, it eliminates token drops, not necessarily perfectly even expert load.
60 second bullets to scan on the way to the call.
Capacity factor buffer sizing
Overflow token drop behavior
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.