Profiling your expert parallel training run shows all to all dispatch and combine eating over a third of step time, and the GPUs sit idle while tokens are in flight. Walk through at least three concrete techniques to hide or shrink that communication cost, and note the tradeoff each one carries.
Three families of fixes: overlap dispatch with compute, route it over faster links (hierarchical or node-limited), or shrink the payload (lower precision, tighter capacity).
Picture a busy office where workers idle while paperwork is couriered between buildings. Three ways to shrink the wait. First, start the next task while the courier is still on the road, so the worker is never idle. Second, use the in-house elevators between floors instead of trucks between buildings, by keeping more paperwork inside the same building. Third, send shorter memos and put a cap on how many envelopes any courier can carry. The first costs scheduling effort, the second costs flexibility in who works on what, and the third risks a worse signal or lost paperwork.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
All to all communication dominates large-scale MoE training step time because the volume scales with batch size, top-k, and hidden dimension simultaneously. A 32-layer model with top-2 routing and 8k hidden moves megabytes of activations per layer per step across the cluster. At scale, this comm time often exceeds the local expert compute it gates.
The interview-grade answer to how you hide that latency decomposes cleanly into three families of mitigation, each attacking a different term in the wallclock equation, each with a real tradeoff. This walkthrough builds those families up, explains the operational ordering you would use in production, and connects the techniques to specific frameworks and frontier models that use them.
Mental model: all to all wallclock equals (bytes over bandwidth) plus scheduling overhead. You can shrink any of the three terms; each technique attacks a different one.
Family one: overlap dispatch with compute
The naive schedule does dispatch, then expert compute, then combine, sequentially. The GPU sits idle while the dispatch is in flight, then again while the combine is in flight. Wallclock is the sum of compute and comm.
Chunked all to all splits the batch into smaller microchunks and pipelines them. While the dispatch of chunk k+1 is in flight over the network, the GPU is already computing the expert FFN for chunk k. Combine of chunk k overlaps with dispatch of chunk k+2. In the limit, wallclock approaches max(compute, comm) instead of compute + comm.
Implementation lives in fused dispatch kernels: Megablocks, Tutel, DeepSpeed-MoE. The tradeoff is two-fold. Activation memory grows because multiple chunks must be resident simultaneously. Scheduling logic is non-trivial under PyTorch's stream model, and getting overlap right requires careful kernel ordering and stream synchronisation.
This is typically the first mitigation teams introduce because it has no quality risk: the model sees identical computation, just scheduled differently. The cost is engineering time and a bit more activation memory.
A concrete sketch. With 4 microchunks per layer and dispatch time roughly equal to expert compute time, the sequential schedule takes . The pipelined schedule reaches a steady state around . That is a 2x to 2.5x step-time win on layers where all to all dominates. The gain shrinks if compute is much larger than comm (no overhead to hide) or much smaller (overlap cannot fully cover comm).
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSpeed-MoE and Megablocks implement chunked all to all with dispatch-compute overlap as the default kernel path.
- DeepSeek-V3 uses node-limited routing to cap cross-node traffic in expert-parallel training at scale.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide between hierarchical all to all and node-limited routing?
Hierarchical preserves expert freedom but adds kernel complexity; node-limited adds a routing constraint but is simpler to implement. Choose based on whether quality regression from routing constraints is acceptable.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Naming one technique repeatedly with different labels. Senior answer covers three distinct families: overlap, topology, payload reduction. Each addresses a different scaling axis.
60 second bullets to scan on the way to the call.
Why all to all volume scales with batch x top-k x hidden dimension
How chunked all to all overlaps dispatch with compute
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.