DDP vs FSDP vs ZeRO
Three ways to shard distributed LLM training across GPUs
DDP replicates the model per GPU; FSDP and ZeRO shard it, freeing memory so bigger models fit. ZeRO is a spec (stages 1-3); FSDP is a fully-shardable PyTorch implementation of the same idea.
DDP
Glossary →DistributedDataParallel replicates the full model on every GPU and averages gradients each step. Simple and fast until the model no longer fits in one GPU's memory.
Best for: Models that comfortably fit in a single GPU.
FSDP
Glossary →Fully Sharded Data Parallel partitions parameters, gradients, and optimizer states across ranks; layers are all-gathered just in time for compute and freed after. PyTorch's native implementation of ZeRO-3.
Best for: Large models that don't fit under DDP.
ZeRO
Glossary →DeepSpeed's Zero Redundancy Optimizer. Stage 1 shards optimizer states, Stage 2 also shards gradients, Stage 3 also shards parameters. Same core idea as FSDP with more knobs (offload, activation checkpointing).
Best for: Fine-grained memory tuning and CPU/NVMe offload.
At a glance
| Dimension | DDP | FSDP | ZeRO |
|---|---|---|---|
| What is sharded | Nothing (full replica) | Params + grads + optimizer states | Optimizer states (Stage 1), + grads (Stage 2), + params (Stage 3) |
| Per-GPU memory | Full model | 1/N of everything | 1/N (Stage 3), or partial by stage |
| Communication | All-reduce on grads | All-gather params + reduce-scatter grads | Similar to FSDP at Stage 3 |
| Ecosystem | PyTorch core | PyTorch native | DeepSpeed |
| Offload support | No | Basic CPU offload | Rich: CPU + NVMe offload flags |
| Best for | Small models, high throughput | Large LLMs on PyTorch stack | Memory tuning + offload for extreme scale |
Key differences
- 1DDP replicates the whole model; FSDP and ZeRO shard it
- 2FSDP is PyTorch-native; ZeRO is DeepSpeed-native (works with PyTorch)
- 3ZeRO Stage 1 shards optimizer states, Stage 2 adds gradients, Stage 3 adds parameters (that is FSDP-equivalent)
- 4FSDP and ZeRO both trade communication for memory: more all-gathers, smaller per-GPU footprint
- 5ZeRO ships CPU / NVMe offload flags; FSDP has offload_to_cpu but simpler
In the interview
- Saying FSDP replaces DDP for all workloads (DDP still wins on small models for throughput)
- Treating ZeRO and FSDP as unrelated when they're the same idea
- Ignoring the communication overhead FSDP and ZeRO add
- Recommending Stage 3 when Stage 1 or 2 already fits
How to choose
Fits in one GPU → DDP. PyTorch-native sharding → FSDP. Offload knobs and stage tuning → ZeRO.
Common misconceptions
Myth: FSDP is a different algorithm from ZeRO.
Reality: FSDP is the PyTorch-native implementation of the ZeRO-3 idea. Same trade: more communication for less memory.
Myth: You should always use Stage 3 or FSDP.
Reality: If DDP fits, DDP wins on throughput. Only shard when memory forces the trade.
Memory aid
DDP: everyone brings their own textbook. FSDP/ZeRO: the class shares one textbook and passes chapters around when needed.
Can you combine them?
In practice you pick one strategy per run. FSDP and ZeRO can be layered with activation checkpointing and mixed precision for further memory savings.
Quiz yourself
Related topics
Related comparisons
Distillation vs Quantization
Two orthogonal ways to shrink a large language model
DPO vs PPO
Direct Preference Optimization vs Proximal Policy Optimization for RLHF
In-context learning vs Fine-tuning
Adapt in the prompt, or adapt in the weights
LoRA vs Full Fine-tuning
Parameter-efficient adaptation vs updating all model weights
RAG vs Fine-tuning
Two approaches to giving LLMs domain specific knowledge