In a large MoE like **DeepSeek-V3**, hundreds of experts are spread across many GPUs, organized into nodes (typically 8 GPUs per node). Within a node, GPUs talk over NVLink at hundreds of GB/s. Between nodes, they talk over InfiniBand or RoCE at tens of GB/s. The intra-node link is roughly an order of magnitude faster.
DeepSeek-V3 routes each token to about 8 experts. If those 8 experts can be anywhere in the cluster, the token has to traverse the slow inter-node fabric for almost all of them. Across the whole batch, this saturates the slow links and step time stalls.
**Node-limited routing** is a rule baked into the gate: when picking experts for a token, you may only pick from at most M nodes (DeepSeek-V3 uses M=4). That bounds the inter-node fan-out per token. Most dispatch stays on NVLink, and only the bounded slice goes over the slow fabric. It is a systems constraint, not a quality or memory trick.
**Node-limited routing** is a hardware-aware constraint on the routing rule. The point is to keep all to all communication concentrated on the fast intra-node interconnect.
The setup. In an expert-parallel cluster, experts are pinned to GPUs and GPUs are grouped into nodes. Intra-node bandwidth (NVLink, NVSwitch) is roughly 5 to 10 times faster than inter-node bandwidth (InfiniBand, RoCE). When a token routes to k experts spread randomly across N nodes, the all to all step has to ship that token's hidden state to all the right nodes, do the FFN computation, and ship results back.
Without the constraint, with k=8 and many nodes, almost every token crosses the slow fabric. The cross-node all to all becomes the step bottleneck.
The constraint. DeepSeek-V3 restricts each token's k selected experts to lie in at most M=4 nodes out of the full cluster. The gate score first picks the top M nodes (by sum of top-2 expert affinities per node), then picks the top k experts within those nodes. The math now puts at most M nodes per token in the all to all pattern, so cross-node volume is bounded by M times batch size, not k times batch size, and most dispatch stays on NVLink.
It is not about memory (experts are still pinned to whichever GPU holds them), not about specialization (specialization comes from training dynamics), and not about determinism (token routing is still data-dependent).
Quick reference
TL;DR
Node-limited routing forces a token's selected experts to live on at most M nodes, so most all to all stays on fast intra-node links and cross-node fabric traffic is bounded by construction.
Imagine a giant office building. Within one floor, people can hand notes to each other instantly. Between floors, notes have to go through a slower elevator. If you write a memo that needs eight signatures from random offices, the elevator becomes a traffic jam. The trick is to require that your memo's signatures all live on at most two floors. The first floor can pass notes quickly among themselves, the second floor can do the same, and only a small bounded amount of paperwork rides the slow elevator. That is what node-limited routing does for a Mixture of Experts model spread across a cluster.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Your cluster fits within a single node (8 GPUs) There is no inter-node fabric to bound; all expert-parallel dispatch already rides NVLink
Your MoE has very few experts (4 to 16) and low top-k All to all volume is small relative to compute; the constraint adds gate complexity for little gain
Inter-node bandwidth approaches intra-node bandwidth (next-gen NVLink fabrics across racks) The asymmetry that motivates the constraint shrinks, and the regularization cost may not be worth it
Quick scan · 2–4 min
Everything important, quickly.
7 minutes: ground the asymmetry between NVLink and InfiniBand, explain expert-parallel all to all, describe the gate-level constraint and how M bounds cross-node volume, and contrast with placement and scheduler tricks.
Real-world examples
Real products, models, and research that use this idea.
1DeepSeek-V3 uses M=4 node-limited routing across 256 experts to keep training communication-bound on intra-node links
2Google's GShard introduced expert-parallel all to all but did not have node-limit awareness, motivating later designs
3DBRX from Databricks uses fewer larger experts to sidestep the all to all cost at training scale
4Megablocks and DeepSpeed-MoE in Megatron now expose node-limited routing as a configurable option for large-cluster MoE pretraining
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does node-limited routing interact with the auxiliary load-balancing loss?
Sign in or go Pro to see the approach for this follow-up.
QCould you achieve the same effect with hierarchical all to all at the communication layer instead of changing the routing rule?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reading node-limited routing as a memory or specialization mechanism, when it is purely a communication-bandwidth constraint baked into the routing rule.