How does dropless MoE dispatch differ from capacity-factor token dropping?

In expert-parallel MoE training, each GPU holds some experts and receives tokens routed to them. Hardware and kernels work best with bounded batch sizes per expert, you cannot send unbounded tokens to one expert in a single step. **Capacity-factor training** sets a fixed buffer per expert (capacity = factor × expected tokens per expert). When more tokens route to an expert than the buffer allows, overflow tokens are **dropped**, they skip the expert FFN and pass through via the residual connection only. This saves compute and bounds worst-case step time at the cost of some tokens never receiving expert transformation that step. **Dropless MoE** pursues **zero token drops**. Dispatch strategies, dynamic buffer sizing, better token batching, redistribution across steps, or optimized all to all scheduling, ensure every routed token is processed by its selected expert. The tradeoff is more complex scheduling and less predictable per-step FLOPs.

Both approaches solve expert overload during training. They differ on whether overflow tokens are allowed to skip FFN compute. **Capacity factor:** fixed per-expert buffer sized as `capacity_factor × (tokens / experts)`. Overflow → token drop or residual-only pass. This is an explicit quality-efficiency tradeoff. Some tokens miss expert compute that step. Step time stays bounded and implementation stays simple (Switch/GShard heritage). **Dropless MoE:** dispatch layer guarantees every routed token reaches its expert FFN, no overflow skips. Achieved via smarter batching, dynamic buffers, pipelined all to all, or token redistribution. Pays scheduling complexity and potentially higher variance in per-step compute. Neither eliminates routing imbalance, dropless removes drops, not skew. You still monitor per-expert load; you just refuse to let overload manifest as skipped tokens.