A 70B full fine-tune in bf16 with Adam needs ~84 GB per GPU if replicated. Only ZeRO-3 or FSDP shard params, grads, and optimizer states to fit on 80 GB H100s.
Picture eight movers carrying one enormous wardrobe up the stairs. With DDP, every mover is told to carry the whole wardrobe alone, so none of them can lift it. ZeRO-1 lets them share only the toolbox, but each still hauls the full wardrobe, still too heavy. ZeRO-3 and FSDP finally split the wardrobe itself into eight slices, so each mover carries one slice and they pass slices around only when a slice is actually needed at the top. That sharing is what makes the load fit on each person. The trade is more talking between movers, which costs a little time, but now the job actually gets done.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question is a memory-arithmetic problem wearing a strategy-selection costume. The answer choices all sound plausible because each is a real, widely used parallelism technique. The way to separate them is not to reason about elegance or popularity. You compute the per-GPU footprint of a 70B full fine-tune and check it against the 80 GB on each H100.
The single decision rule for any multi-GPU fine-tune is this: does one complete copy of the training state fit on one device? The training state is not just the weights. For a full fine-tune with Adam, it is weights plus gradients plus two optimizer moments. If that copy fits, you can replicate it and use simple data parallelism. If it does not fit, you must shard the state itself across devices, and that is exactly what ZeRO-3 and FSDP do.
This deep dive does the arithmetic explicitly, shows why each wrong option overflows, explains how full sharding gathers and frees weights on demand, and then covers the communication cost and the move to 3D parallelism that production runs make at larger scale.
The memory arithmetic that decides everything
Start with the four consumers of GPU memory in a full fine-tune: parameters, gradients, optimizer states, and activations. For a 70B model in bf16, each parameter takes 2 bytes, so the live weights are about 140 GB total and the gradients are another 140 GB total. Those two alone already dwarf a single 80 GB card, which is the first signal that naive replication is hopeless.
Adam is the expensive part. The standard mixed-precision recipe keeps an fp32 master copy of the weights plus two fp32 moment buffers, the first and second moments. That is roughly 12 bytes per parameter for optimizer-related state, or about 840 GB total across the model. People often quote the famous 16-bytes per parameter figure, which folds the bf16 weights, bf16 gradients, and the fp32 Adam state into one round number for a full fine-tune.
The interview-relevant move is to convert totals into a per-GPU number and compare against 80 GB. With 8 cards, the optimizer state divides cleanly only if you actually shard it. If you replicate, every GPU carries the full weights and gradients in bf16 at 14 GB each, plus the optimizer-related state, which alone pushes the parameter-side total to roughly 84 GB per card.
That 84 GB is the headline figure, and it is computed before a single activation tensor is allocated. The conclusion is immediate. Any strategy that keeps the weights replicated cannot fit, so the only viable family is one that shards the weights themselves.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Strategy | What it shards | Fits 70B full FT on 80 GB? |
|---|---|---|
| DDP | Nothing; full replica per GPU | No, ~84 GB replicated state per card |
| ZeRO-1 | Optimizer states only | No, weights plus grads still ~28 GB plus activations |
| ZeRO-3 / FSDP | Parameters, gradients, optimizer states | Yes, ~12 GB params per card plus headroom |
| Pipeline alone | Layers across stages | No, each stage holds full Adam state for its layers |
Real products, models, and research that use this idea.
- PyTorch FSDP is the native path for full fine-tunes of Llama 3.1 70B on 8x H100 nodes, sharding all training state across ranks.
- DeepSpeed ZeRO-3 powers large open-model fine-tunes and underlies many runs that produced DeepSeek V4 and similar frontier-scale checkpoints.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does FSDP's all-gather and reduce-scatter pattern change communication volume compared with DDP's gradient all-reduce?
Walk through what crosses the wire per layer. DDP all-reduces gradients once per step; FSDP all-gathers weights every layer in forward and backward, then reduce-scatters gradients, raising volume but cutting memory.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for DDP or ZeRO-1 on a 70B full fine-tune because they are simpler, then hitting out of memory because the replicated weights alone exceed 80 GB per card.
60 second bullets to scan on the way to the call.
Per-GPU memory budget for a full fine-tune in bf16 with Adam
Why weights, gradients, and optimizer states each cost separately
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.