The right answer is C: **ZeRO-3 or FSDP**. The reason is pure memory arithmetic, not preference.
A full fine-tune of a 70B model in bf16 with the Adam optimizer needs three big things in memory: the weights, the gradients, and the optimizer states. Add them up and you get roughly 84 GB just for the parameters. An H100 only has 80 GB, so a single copy does not even fit.
DDP gives every GPU a full copy of the model, so it never fits a 70B. ZeRO-1 shards only the Adam optimizer states, but the weights and gradients stay replicated, which still busts the budget. Pipeline parallelism alone splits layers across GPUs, yet each stage still holds full optimizer state for its layers, so it also overflows.
ZeRO-3 and FSDP shard all three pieces across the eight GPUs. Each card now holds about one-eighth of the parameters and gathers the rest only when a layer runs. That drops the per-GPU footprint low enough to fit, leaving room for activations.
The answer is ZeRO-3 or FSDP, and the deciding factor is whether one full copy of the training state fits on a single GPU.
For a 70B full fine-tune in bf16 with Adam, you carry weights, gradients, and two optimizer moments. That is roughly 14 plus 14 plus 56, about 84 GB per GPU if replicated. An H100 holds 80 GB, so any strategy that replicates weights is dead on arrival.
That rules out DDP, which replicates everything, and ZeRO-1, which shards only optimizer states while weights and gradients stay replicated. Pipeline parallelism alone also fails here. Each stage still stores full Adam state for its slice of layers, so eight stages of a 70B still overflow.
ZeRO-3 and FSDP are the same idea from two ecosystems. They shard parameters, gradients, and optimizer states across all eight cards. Each GPU keeps one-eighth and all-gathers a layer's weights only for the moment that layer runs, then frees them. Per-GPU parameter memory drops to roughly 12 GB, leaving headroom for activations. The cost is more communication, which is why production runs often layer tensor parallelism on top.
Quick reference
TL;DR
A 70B full fine-tune in bf16 with Adam needs ~84 GB per GPU if replicated. Only ZeRO-3 or FSDP shard params, grads, and optimizer states to fit on 80 GB H100s.
Picture eight movers carrying one enormous wardrobe up the stairs. With DDP, every mover is told to carry the whole wardrobe alone, so none of them can lift it. ZeRO-1 lets them share only the toolbox, but each still hauls the full wardrobe, still too heavy. ZeRO-3 and FSDP finally split the wardrobe itself into eight slices, so each mover carries one slice and they pass slices around only when a slice is actually needed at the top. That sharing is what makes the load fit on each person. The trade is more talking between movers, which costs a little time, but now the job actually gets done.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Small model or LoRA fine-tune where one full copy of training state fits comfortably on a single GPU Sharding adds communication overhead with no benefit. Plain DDP is faster and simpler when memory is not the constraint.
A workload bottlenecked by inter-node bandwidth rather than per-GPU memory capacity Full parameter sharding multiplies all-gather traffic. If bandwidth is the limit, tensor parallelism inside a node may scale better.
Tiny cluster where even fully sharded parameters plus activations exceed total aggregate memory Sharding cannot create capacity that does not exist. You need more GPUs, offload to CPU or NVMe, or a smaller base model.
Prototyping logic on a 1 to 7B model before scaling the recipe up The memory pressure that justifies ZeRO-3 is absent at small scale. Add sharding only when the per-GPU budget actually breaks.
Quick scan · 2–4 min
Everything important, quickly.
5 min: do the per-GPU memory math, rule out DDP and ZeRO-1 and pipeline-alone, explain how ZeRO-3 and FSDP shard all three state types, then cover the communication cost and 3D parallelism.
Side-by-side
Strategy
What it shards
Fits 70B full FT on 80 GB?
DDP
Nothing; full replica per GPU
No, ~84 GB replicated state per card
ZeRO-1
Optimizer states only
No, weights plus grads still ~28 GB plus activations
ZeRO-3 / FSDP
Parameters, gradients, optimizer states
Yes, ~12 GB params per card plus headroom
Pipeline alone
Layers across stages
No, each stage holds full Adam state for its layers
Real-world examples
Real products, models, and research that use this idea.
1PyTorch FSDP is the native path for full fine-tunes of Llama 3.1 70B on 8x H100 nodes, sharding all training state across ranks.
2DeepSpeed ZeRO-3 powers large open-model fine-tunes and underlies many runs that produced DeepSeek V4 and similar frontier-scale checkpoints.
3Hugging Face Accelerate exposes both FSDP and DeepSpeed ZeRO-3 behind one config so teams swap sharding backends without rewriting the trainer.
4Megatron-LM combines tensor parallelism, pipeline parallelism, and ZeRO-style sharding into 3D parallelism for the largest training and fine-tuning runs.
5NVIDIA NeMo reference recipes layer FSDP or ZeRO-3 with tensor parallelism on DGX H100 systems for full fine-tunes above 70B.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does FSDP's all-gather and reduce-scatter pattern change communication volume compared with DDP's gradient all-reduce?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you add tensor parallelism on top of FSDP instead of just scaling FSDP across more nodes?
Sign in or go Pro to see the approach for this follow-up.
QHow do activation checkpointing and CPU or NVMe offload interact with ZeRO-3 to push fine-tunes past the GPU memory wall?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reaching for DDP or ZeRO-1 on a 70B full fine-tune because they are simpler, then hitting out of memory because the replicated weights alone exceed 80 GB per card.