Match multi-GPU training strategy to what it shards
Drag each answer to line up with its matching prompt
DDP
Shards optimizer states and gradients; weights are still replicated.
ZeRO-1
Replicates the full model on every GPU and synchronises gradients via all-reduce after each backward pass.
ZeRO-2
Splits individual weight matrices across GPUs, requiring custom all-reduce inside the forward/backward of attention and MLP.
ZeRO-3
Shards only the optimizer states across GPUs; weights and gradients are still replicated on every GPU.
FSDP
PyTorch's native equivalent of ZeRO-3: parameters, gradients, and optimizer states all sharded.
Tensor parallelism
Shards optimizer states, gradients, AND weights: each GPU holds only a slice of the parameters.
DDP replicates and all-reduces gradients. ZeRO-1/2/3 progressively shard optimizer, gradients, then weights. FSDP is PyTorch ZeRO-3. Tensor parallelism cuts each matrix.
Imagine a team copying one big recipe book to cook a huge meal. With DDP, every cook owns a full copy and they sync notes after each dish. That works until the book is too heavy to hold. ZeRO is a librarian who says, you do not all need the full book at once. First everyone shares the bookmarks, then the scribbled corrections, then finally the pages themselves, so each cook stores only a slice and borrows pages when needed. FSDP is the same trick built into PyTorch. The matrix-splitting approach is different: a single page is so wide that two cooks read the left and right halves at the same time, then combine their reading. You pick the trick based on whether the book fits on one shelf.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-GPU training questions look like trivia until you see the single organising principle behind them. Every strategy on this list answers one question: what does each GPU need to store, and what must it communicate to stay correct? Match the strategy to the memory pressure and the answer falls out cleanly. The names DDP, ZeRO, FSDP, and tensor parallelism stop being a vocabulary list and start being points on two clean axes.
The confusion in interviews comes from treating these as a flat menu of interchangeable options. They are not. Two of them, data parallelism and model parallelism, are completely different axes. DDP and the ZeRO family are all data parallelism: every rank works on the same parameters, the only difference is whether each rank keeps a full copy or a shard. Tensor parallelism is model parallelism: a single matrix is physically split across ranks and no rank ever holds the whole thing. A strong answer always separates these two ideas before naming any specific technique.
The second thing a strong answer makes explicit is that none of this is free. Memory and communication trade against each other. DDP buys minimal communication at the cost of full replication. ZeRO and FSDP buy huge memory savings by adding all-gather and reduce-scatter traffic. Tensor parallelism buys the ability to split a single oversized layer, but pays with all-reduce on the critical path of every block. There is no universally best choice, only the right match to the constraint that is actually binding.
This deep dive walks the ladder. It starts with the memory accounting that drives every decision, covers the DDP baseline, climbs the three ZeRO stages, shows why FSDP is the PyTorch-native ZeRO-3, then crosses over to tensor parallelism and explains how the largest training runs fuse all three axes into 3D parallelism.
The memory footprint that forces the decision
To pick a strategy you first have to know where the memory goes. For a model trained with the Adam optimizer in mixed precision, the per-parameter cost is dominated not by the weights but by the optimizer state. This is the most common surprise for engineers used to thinking only about parameter count.
The rough accounting per parameter is: two bytes for the bf16 weight, two bytes for the bf16 gradient, and twelve bytes of optimizer state. Those twelve bytes are a fp32 master copy of the weight plus the fp32 first and second moments that Adam maintains. That sums to roughly sixteen bytes per parameter, and the optimizer alone is three quarters of it.
That single fact explains the entire ZeRO ladder. The biggest, most redundant memory term is the optimizer state, so it is the first thing worth sharding. Gradients are next, and the parameters themselves are last because sharding them costs the most communication. The ladder is not arbitrary; it is sorted by memory saved per unit of extra communication incurred.
This also explains why activations matter separately. The sixteen-bytes figure covers persistent training state, but the forward pass also stores intermediate activations for backward. Activation memory scales with batch size and sequence length, not parameter count, and is why gradient checkpointing is almost always paired with these sharding strategies.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Strategy | What is sharded | Use when |
|---|---|---|
| DDP | Nothing; full replica per GPU | Model plus optimizer fits one GPU |
| ZeRO-1 | Optimizer states only | Optimizer memory is the bottleneck |
| ZeRO-2 | Optimizer states and gradients | Gradient memory also tight |
| ZeRO-3 / FSDP | Optimizer, gradients, and parameters | Model does not fit replicated |
| Tensor parallel | Each weight matrix split across GPUs | A single layer overflows one GPU |
Real products, models, and research that use this idea.
- Meta trained Llama 4 using PyTorch FSDP combined with tensor and pipeline parallelism across large H100 clusters.
- Microsoft DeepSpeed ZeRO-3 underpins many open-weight fine-tuning runs, including community fine-tunes of DeepSeek V4 and Qwen 3.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does ZeRO-3 add roughly 1.5 times the communication of DDP, and when is that trade worth it?
Count the all-gather of parameters in forward and again in backward plus the gradient reduce-scatter. The trade pays off when memory, not bandwidth, is the binding constraint.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating FSDP and tensor parallelism as the same thing. FSDP shards the list of parameters across data-parallel ranks; tensor parallelism splits each individual weight matrix.
60 second bullets to scan on the way to the call.
The single question that picks a strategy: does the model fit one GPU
What DDP replicates and how it synchronises
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.