Design the parallelism layout for training a 16-expert MoE across 64 GPUs.
You are planning a training run for a 16-expert MoE on 64 GPUs arranged as 8 nodes of 8 GPUs (fast NVLink within a node, slower InfiniBand between nodes). Design the parallelism layout: how do you combine expert, tensor, and data parallelism, where do the communication heavy operations land, and what failure mode do you watch for?
EP=8 inside nodes so all to all rides NVLink, DP=8 across nodes so gradient all-reduce uses InfiniBand, add TP only if memory forces it.
Picture a relay race with two kinds of batons: one you must hand off many times per lap, and one you only swap at the finish line. The first baton wants the fastest runners in the inner lane, that is the all to all token dispatch, and it goes on the NVLink inside each node. The second baton can travel between teams across a slower outdoor stretch, that is the once per step gradient sync, and it goes on InfiniBand between nodes. If a team's locker room is too small for both its uniforms and the baton-handoff staging area, you split the team across two lockers (tensor parallelism), but only inside the same building. Watch out for one runner who keeps getting handed twice the batons of everyone else, because the whole race finishes when the slowest runner finishes.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Designing parallelism for an MoE training run is one of the highest-leverage decisions in the whole project. A wrong layout wastes weeks of GPU time and may not be diagnosable from output quality alone. A right one extracts close to peak hardware utilization. The question's specific topology (8 nodes of 8 GPUs, 64 GPUs total) and model (16 experts) is a clean factorization problem on top of a deep mental model.
This walkthrough builds the model in four steps: the interconnect hierarchy that orders the parallelism dimensions, the canonical layout that falls out, the deviations forced by memory pressure, and the two failure modes that demand monitoring from hour one of the run.
The senior signal: name the interconnect hierarchy first, factorize against it, then handle memory as a deviation rather than as a co-equal constraint. Layouts that start from memory and then patch around communication almost always end up with cross-node all to all somewhere.
The interconnect hierarchy and what each collective costs
NVLink versus InfiniBand
On H100 and H200 nodes, NVLink delivers roughly 600 to 900 GB/s per GPU within the node. InfiniBand HDR/NDR between nodes is 50 to 100 GB/s per node. The bandwidth gap is 10x to 18x. Latency is also worse cross-node: tens of microseconds intra-node, hundreds across nodes.
This is the fundamental constraint that orders the whole design. Any collective that runs frequently and is on the critical path of the forward or backward pass cannot afford to ride InfiniBand. Any collective that runs once per step and can overlap with compute is fine on InfiniBand.
Ranking the collectives in an MoE training run
From most to least latency-sensitive:
- All to all token dispatch (MoE). Fires per MoE layer per forward and again per backward. Hundreds of small messages. Latency-bound. Empirically dominates step time when forced cross-node.
- Tensor-parallel all-reduce. Fires per linear layer per forward and backward. Latency-bound. Megatron-LM documents that TP must stay intra-node.
- Pipeline-parallel send/recv. Fires per micro-batch boundary. point to point not collective. Tolerates inter-node when bubble is bounded.
- Data-parallel gradient all-reduce. Fires once per step. Bandwidth-bound but overlappable with backward compute via bucketed reduction. The most forgiving collective.
What this ordering buys you
Once the ordering is fixed, the layout is no longer a free-choice problem. Latency-sensitive dimensions go on the fast link; tolerant dimensions go on the slow link. The factorization then has to satisfy the constraint that all degrees multiply to the GPU count.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral 8x7B and 8x22B training at Mistral AI uses expert parallelism within nodes and data parallelism across nodes, with TP introduced only for the largest variants where attention memory forces it.
- DeepSeek-V3 publishes its parallelism strategy in the technical report, including node-limited routing to bound cross-node all to all and aux loss-free balancing to remove the gradient-competition issue.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the layout change if the cluster has 4 nodes of 16 GPUs instead of 8 nodes of 8 GPUs?
Wider nodes mean more headroom for EP intra-node. With 16 GPUs per node and 16 experts, EP=16 fits exactly inside one node, one expert per GPU, all to all stays on NVLink. DP=4 across nodes. The trade is that wider nodes have larger NVLink topologies and intra-node all to all may itself need hierarchical dispatch on NVSwitch fabrics.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Spreading expert parallelism across nodes so that the all to all dispatch is forced onto InfiniBand on every forward pass, which dominates step time and dwarfs any gain from a wider degree of EP.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.