DDP vs FSDP for a 13B full fine-tune on 4xA100-40GB: which is feasible?

MCQ·Medium·4.0 · 0·~1 min·Asked atCursor Graphcore Spotify·Relevant atDatabricks

Attempt it

**The answer is FSDP.** DDP cannot fit a 13B full fine-tune on 4xA100-40GB cards, FSDP can. The difference is in what each GPU holds. With DDP (DistributedDataParallel), every GPU keeps a full copy of the model. For 13 billion parameters in bf16 that is roughly 26GB just for the weights. Add gradients (another 26GB), Adam optimizer state (roughly 52GB for the two momentum tensors in fp32), and any activations and you are well past the 40GB limit per card. The training run will simply OOM. With FSDP (FullyShardedDataParallel), the weights, gradients, and optimizer state are split across the four GPUs. Each card holds roughly a quarter of each, so the per-GPU memory budget is roughly a quarter of what DDP would need. The 13B fits comfortably on 40GB cards with FSDP, with headroom for activations and buffer space. The other options miss this. Option (a) is wrong because DDP does not share resident memory, it only shares gradient traffic during the all-reduce. Option (b) is wrong because DDP and FSDP have very different per-GPU footprints, that is the whole point of the split. Option (d) overstates the hardware constraint, FSDP is precisely what makes the 13B feasible on 40GB cards. DDP is the right choice when the model fits comfortably on one GPU and you want maximum throughput. FSDP is the right choice when the model does not fit and you need to shard to make training feasible.

**FSDP is the feasible choice.** DDP cannot fit a 13B full fine-tune on 4xA100-40GB cards; FSDP can. The core distinction is what each GPU resident memory holds. DDP keeps a full replica of the model on every GPU. For 13 billion parameters in bf16, that is roughly 26GB of weights per card. Add gradients (roughly 26GB more, same shape as weights), Adam optimizer state (roughly 104GB for fp32 momentum and variance at 8 bytes per parameter), plus activations and miscellaneous buffers, and the per-GPU budget needs well over 100GB. A 40GB card cannot hold this, training OOMs immediately. FSDP shards parameters, gradients, and optimizer state across all four GPUs. Each card holds roughly a quarter of each tensor. The per-GPU memory budget drops to roughly a quarter of what DDP would need, which fits on 40GB with headroom for activations and communication buffers. The trade-off is that FSDP has to all-gather shards on the fly during forward and backward passes, which adds communication overhead, but on a fast NVLink or InfiniBand network this is manageable. **Why the other options are wrong.** Option (a) misunderstands DDP. DDP does not share resident memory across GPUs, it only all-reduces gradients during the backward pass. The replicated model still sits on every card. Option (b) is wrong because DDP and FSDP have very different per-GPU memory footprints, the whole reason FSDP exists is to shard what DDP replicates. Option (d) overstates the constraint; 40GB cards can absolutely run 13B full fine-tunes with FSDP, the larger A100-80GB and H100 are needed for full fine-tunes above 30B or for runs that need very long context.

TL;DR

DDP replicates the full 13B on every GPU, which busts 40GB once gradients and Adam state are added. FSDP shards weights, grads, and optimizer state across the 4 GPUs, dropping per-GPU memory to a quarter, and fits.

Memory aid

Easy to grasp

Picture four friends moving house. With the DDP strategy each friend carries a complete copy of every box, so the heaviest single load is the whole apartment, no one can lift it. With the FSDP strategy the boxes are split and each friend carries only a quarter of them, talking to the others when they need something the others are holding. The total weight is the same, but no single friend is crushed. For a 13B model on 40GB cards the difference between crushed and fine is exactly that.

Key concepts

FSDP DDP Distributed training Memory budget

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Choosing between DDP and FSDP is one of the most consequential decisions in a multi-GPU fine-tuning setup. The two strategies look superficially similar (both scale training across multiple GPUs) but have very different memory models, and the difference becomes critical once the model crosses the size where it stops fitting on a single GPU.

The 13B-on-4xA100-40GB scenario in this question is exactly the regime where the choice forces itself. DDP cannot fit the model because it replicates everything on every GPU, and a 13B full fine-tune with Adam in mixed precision needs well over 100GB of resident state per replica. FSDP shards that state across the four GPUs, dropping the per-GPU budget to roughly a quarter of the replicated version, which fits comfortably on 40GB cards.

This deep dive walks the per-GPU memory math, explains how DDP and FSDP differ in what they share and what they replicate, covers when each strategy is the right choice, and connects the picture to related techniques like DeepSpeed ZeRO and gradient checkpointing that interact with FSDP in practical fine-tuning recipes.

The per-GPU memory math for a 13B full fine-tune

A full fine-tune with Adam in mixed precision needs four categories of resident state on each GPU that holds the model.

Weights in bf16 occupy 2 bytes per parameter. For 13 billion parameters that is 26GB.

Gradients in bf16 occupy the same shape and precision as the weights. Another 26GB.

Adam optimizer state in fp32 occupies 8 bytes per parameter (4 bytes for the first moment, 4 bytes for the second moment). For 13 billion parameters that is 104GB. Adam state is in fp32 because the moments need higher precision than the bf16 weights to track update statistics correctly.

Activations and miscellaneous buffers scale with batch size, sequence length, and the activation checkpointing strategy. For a typical 13B fine-tune with moderate batch size and 4k sequence length, expect 5 to 15GB.

The total comes to roughly 160 to 170GB per replica. This is the memory a single GPU would need to hold one complete copy of the training state.

A 40GB A100 obviously cannot hold this. Even an 80GB A100 would not hold it with comfortable headroom. The single-GPU training story for 13B full fine-tunes is essentially impossible at this state size, you need to either shrink the state (PEFT methods, quantization) or shard it across multiple GPUs (FSDP, ZeRO Stage 3).

What DDP replicates and what it shares

How FSDP shards and what it costs

When each strategy wins

FSDP in practice with the modern stack

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

PyTorch FSDP is the standard backbone for 2026 community fine-tunes of Llama 3.1 70B and DeepSeek V4 on consumer or modest enterprise clusters.
Hugging Face Trainer and Accelerate both wrap FSDP behind a config flag, so swapping DDP for FSDP is usually a one-line change to a YAML.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does FSDP relate to DeepSpeed ZeRO Stages 1, 2, and 3?

ZeRO Stage 1 shards optimizer state only (closest to DDP plus optimizer sharding). Stage 2 shards optimizer state and gradients. Stage 3 shards optimizer state, gradients, and parameters, which is functionally equivalent to FSDP's full-shard mode. The two stacks differ in implementation details and ergonomics but have similar memory profiles.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing DDP shards memory across GPUs. It does not. DDP replicates the full model on every GPU and only shards the all-reduce traffic during the backward pass, the resident memory per GPU is the same as single-GPU training.

60 second bullets to scan on the way to the call.

What DDP replicates per GPU and what it shares
What FSDP shards per GPU and what communication it adds

Primary sources. Browse if you want the original framing.