Multi-GPU FT: pick the right sharding strategy for a 70B full FT on 8 H100s

The right answer is C: **ZeRO-3 or FSDP**. The reason is pure memory arithmetic, not preference. A full fine-tune of a 70B model in bf16 with the Adam optimizer needs three big things in memory: the weights, the gradients, and the optimizer states. Add them up and you get roughly 84 GB just for the parameters. An H100 only has 80 GB, so a single copy does not even fit. DDP gives every GPU a full copy of the model, so it never fits a 70B. ZeRO-1 shards only the Adam optimizer states, but the weights and gradients stay replicated, which still busts the budget. Pipeline parallelism alone splits layers across GPUs, yet each stage still holds full optimizer state for its layers, so it also overflows. ZeRO-3 and FSDP shard all three pieces across the eight GPUs. Each card now holds about one-eighth of the parameters and gathers the rest only when a layer runs. That drops the per-GPU footprint low enough to fit, leaving room for activations.

The answer is ZeRO-3 or FSDP, and the deciding factor is whether one full copy of the training state fits on a single GPU. For a 70B full fine-tune in bf16 with Adam, you carry weights, gradients, and two optimizer moments. That is roughly 14 plus 14 plus 56, about 84 GB per GPU if replicated. An H100 holds 80 GB, so any strategy that replicates weights is dead on arrival. That rules out DDP, which replicates everything, and ZeRO-1, which shards only optimizer states while weights and gradients stay replicated. Pipeline parallelism alone also fails here. Each stage still stores full Adam state for its slice of layers, so eight stages of a 70B still overflow. ZeRO-3 and FSDP are the same idea from two ecosystems. They shard parameters, gradients, and optimizer states across all eight cards. Each GPU keeps one-eighth and all-gathers a layer's weights only for the moment that layer runs, then frees them. Per-GPU parameter memory drops to roughly 12 GB, leaving headroom for activations. The cost is more communication, which is why production runs often layer tensor parallelism on top.