Node-limited routing forces a token's selected experts to live on at most M nodes, so most all to all stays on fast intra-node links and cross-node fabric traffic is bounded by construction.
Imagine a giant office building. Within one floor, people can hand notes to each other instantly. Between floors, notes have to go through a slower elevator. If you write a memo that needs eight signatures from random offices, the elevator becomes a traffic jam. The trick is to require that your memo's signatures all live on at most two floors. The first floor can pass notes quickly among themselves, the second floor can do the same, and only a small bounded amount of paperwork rides the slow elevator. That is what node-limited routing does for a Mixture of Experts model spread across a cluster.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Training a 600B-parameter MoE on thousands of GPUs exposes a systems problem that dense models do not have. The MoE layer requires two all to all collectives per forward pass: one to dispatch tokens to their assigned experts, and one to combine the FFN outputs back. With dozens of experts and many nodes, naive routing makes those collectives dominate step time.
This deep dive explains why all to all is on the critical path, derives the bandwidth math that motivates the node limit, walks through how DeepSeek-V3 implements the constraint at the gate, and contrasts it with other layers of the stack that could reduce the same cost.
Why all to all dominates MoE step time
In a dense transformer, the dominant collective is gradient all-reduce at the end of each step. All-reduce overlaps well with backward compute, so it rarely gates step time on a well-tuned cluster.
MoE is different. Each MoE layer fires two all to all collectives in the forward pass and two more in the backward pass. Dispatch ships every token's hidden state to the GPUs holding its k selected experts. Combine ships the FFN outputs back to the originating GPU. Unlike all-reduce, these are on the forward critical path. They cannot hide behind compute the same way.
The data volume scales with batch size, sequence length, hidden dim, and k. For DeepSeek-V3 with hidden dim 7168, top-8 routed plus 1 shared expert, and a multi-thousand token batch, the per-step all to all volume runs into many GB per MoE layer. With 60+ MoE layers and two all to alls per layer, the total per step is large enough that even fast interconnect spends meaningful wall-clock time on it.
The bandwidth limit on the link decides whether this all to all becomes the step bottleneck. If most of the volume rides NVLink at hundreds of GB/s, step time stays compute-bound. If most rides InfiniBand at tens of GB/s, the collective gates everything.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V3 uses M=4 node-limited routing across 256 experts to keep training communication-bound on intra-node links
- Google's GShard introduced expert-parallel all to all but did not have node-limit awareness, motivating later designs
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does node-limited routing interact with the auxiliary load-balancing loss?
DeepSeek-V3 layers a device-balance loss on top of expert-balance so both expert and node utilization stay even. The two constraints compose without conflict because they regularize different aggregates of the gate output.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading node-limited routing as a memory or specialization mechanism, when it is purely a communication-bandwidth constraint baked into the routing rule.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.