Match multi-GPU training strategy to what it shards
DDP replicates and all-reduces gradients. ZeRO-1/2/3 progressively shard optimizer, gradients, then weights. FSDP is PyTorch ZeRO-3. Tensor parallelism cuts each matrix.
Imagine a team copying one big recipe book to cook a huge meal. With DDP, every cook owns a full copy and they sync notes after each dish. That works until the book is too heavy to hold. ZeRO is a librarian who says, you do not all need the full book at once. First everyone shares the bookmarks, then the scribbled corrections, then finally the pages themselves, so each cook stores only a slice and borrows pages when needed. FSDP is the same trick built into PyTorch. The matrix-splitting approach is different: a single page is so wide that two cooks read the left and right halves at the same time, then combine their reading. You pick the trick based on whether the book fits on one shelf.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: data vs model parallel framing + DDP baseline + ZeRO 1/2/3 ladder + FSDP as ZeRO-3 + tensor parallelism at the matrix level + how 3D parallelism composes the axes.
| Strategy | What is sharded | Use when |
|---|---|---|
| DDP | Nothing; full replica per GPU | Model plus optimizer fits one GPU |
| ZeRO-1 | Optimizer states only | Optimizer memory is the bottleneck |
| ZeRO-2 | Optimizer states and gradients | Gradient memory also tight |
| ZeRO-3 / FSDP | Optimizer, gradients, and parameters | Model does not fit replicated |
| Tensor parallel | Each weight matrix split across GPUs | A single layer overflows one GPU |
Real products, models, and research that use this idea.
- Meta trained Llama 4 using PyTorch FSDP combined with tensor and pipeline parallelism across large H100 clusters.
- Microsoft DeepSpeed ZeRO-3 underpins many open-weight fine-tuning runs, including community fine-tunes of DeepSeek V4 and Qwen 3.
- NVIDIA Megatron-LM popularised tensor parallelism for splitting attention and MLP matrices within a node over NVLink.
- Hugging Face Accelerate exposes both DeepSpeed ZeRO and native FSDP as one-flag backends for fine-tuning jobs.
- Mosaic and Databricks reference stacks default to FSDP for billion-parameter fine-tunes and layer in tensor parallelism only when a single layer overflows.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does ZeRO-3 add roughly 1.5 times the communication of DDP, and when is that trade worth it?
QWhy are tensor-parallel groups kept inside a single node while pipeline parallelism spans nodes?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating FSDP and tensor parallelism as the same thing. FSDP shards the list of parameters across data-parallel ranks; tensor parallelism splits each individual weight matrix.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.