DDP vs FSDP for a 13B full fine-tune on 4xA100-40GB: which is feasible?

MCQ·Medium·4.0 · 0·~1 min

Attempt it

**The answer is FSDP.** DDP cannot fit a 13B full fine-tune on 4xA100-40GB cards, FSDP can. The difference is in what each GPU holds. With DDP (DistributedDataParallel), every GPU keeps a full copy of the model. For 13 billion parameters in bf16 that is roughly 26GB just for the weights. Add gradients (another 26GB), Adam optimizer state (roughly 52GB for the two momentum tensors in fp32), and any activations and you are well past the 40GB limit per card. The training run will simply OOM. With FSDP (FullyShardedDataParallel), the weights, gradients, and optimizer state are split across the four GPUs. Each card holds roughly a quarter of each, so the per-GPU memory budget is roughly a quarter of what DDP would need. The 13B fits comfortably on 40GB cards with FSDP, with headroom for activations and buffer space. The other options miss this. Option (a) is wrong because DDP does not share resident memory, it only shares gradient traffic during the all-reduce. Option (b) is wrong because DDP and FSDP have very different per-GPU footprints, that is the whole point of the split. Option (d) overstates the hardware constraint, FSDP is precisely what makes the 13B feasible on 40GB cards. DDP is the right choice when the model fits comfortably on one GPU and you want maximum throughput. FSDP is the right choice when the model does not fit and you need to shard to make training feasible.

**FSDP is the feasible choice.** DDP cannot fit a 13B full fine-tune on 4xA100-40GB cards; FSDP can. The core distinction is what each GPU resident memory holds. DDP keeps a full replica of the model on every GPU. For 13 billion parameters in bf16, that is roughly 26GB of weights per card. Add gradients (roughly 26GB more, same shape as weights), Adam optimizer state (roughly 104GB for fp32 momentum and variance at 8 bytes per parameter), plus activations and miscellaneous buffers, and the per-GPU budget needs well over 100GB. A 40GB card cannot hold this, training OOMs immediately. FSDP shards parameters, gradients, and optimizer state across all four GPUs. Each card holds roughly a quarter of each tensor. The per-GPU memory budget drops to roughly a quarter of what DDP would need, which fits on 40GB with headroom for activations and communication buffers. The trade-off is that FSDP has to all-gather shards on the fly during forward and backward passes, which adds communication overhead, but on a fast NVLink or InfiniBand network this is manageable. **Why the other options are wrong.** Option (a) misunderstands DDP. DDP does not share resident memory across GPUs, it only all-reduces gradients during the backward pass. The replicated model still sits on every card. Option (b) is wrong because DDP and FSDP have very different per-GPU memory footprints, the whole reason FSDP exists is to shard what DDP replicates. Option (d) overstates the constraint; 40GB cards can absolutely run 13B full fine-tunes with FSDP, the larger A100-80GB and H100 are needed for full fine-tunes above 30B or for runs that need very long context.

Quick reference

TL;DR

DDP replicates the full 13B on every GPU, which busts 40GB once gradients and Adam state are added. FSDP shards weights, grads, and optimizer state across the 4 GPUs, dropping per-GPU memory to a quarter, and fits.

Memory aid

Easy to grasp

Picture four friends moving house. With the DDP strategy each friend carries a complete copy of every box, so the heaviest single load is the whole apartment, no one can lift it. With the FSDP strategy the boxes are split and each friend carries only a quarter of them, talking to the others when they need something the others are holding. The total weight is the same, but no single friend is crushed. For a 13B model on 40GB cards the difference between crushed and fine is exactly that.

Key concepts

FSDP DDP Distributed training Memory budget

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: per-GPU memory math for 13B with Adam in mixed precision + DDP's replication model + FSDP's sharding model + why DDP busts the 40GB budget + why FSDP fits + when DDP still wins + relation to ZeRO Stages and gradient checkpointing.

Real-world examples

Real products, models, and research that use this idea.

PyTorch FSDP is the standard backbone for 2026 community fine-tunes of Llama 3.1 70B and DeepSeek V4 on consumer or modest enterprise clusters.
Hugging Face Trainer and Accelerate both wrap FSDP behind a config flag, so swapping DDP for FSDP is usually a one-line change to a YAML.
DeepSpeed ZeRO Stage 3 is functionally equivalent to FSDP for memory sharding, the two are interchangeable in most practical contexts.
Axolotl and LLaMA-Factory default to FSDP for full fine-tunes above 7B in their 2026 reference recipes for exactly this memory reason.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does FSDP relate to DeepSpeed ZeRO Stages 1, 2, and 3?

QWhat changes if you add gradient checkpointing on top of FSDP?

QWhy does FSDP need to all-gather shards just in time rather than keeping the full model materialised?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing DDP shards memory across GPUs. It does not. DDP replicates the full model on every GPU and only shards the all-reduce traffic during the backward pass, the resident memory per GPU is the same as single-GPU training.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.