FSDP (Fully Sharded Data Parallel)
Also known as: Fully Sharded Data Parallel, ZeRO-3
Shard weights+grads+optimizer states across GPUs and gather just-in-time; train huge models with limited per-GPU memory.
A data-parallel training strategy that shards model weights, gradients, and optimizer states across GPUs and gathers parameters just-in-time per forward/backward pass. PyTorch's open implementation of the ZeRO-3 algorithm.
In practice
The default for open-source large-model training. Senior training interviews probe FSDP vs DeepSpeed ZeRO and the bandwidth trade-offs.
How it compares
FSDP shards data-parallel state (weights, grads, opt); TP shards model compute within layers.
Comparisons that include FSDP (Fully Sharded Data Parallel)
Related topics
Related terms
DPO (Direct Preference Optimization)
Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.
Fine-tuning
Continue training a pre-trained model on task-specific data to specialize its behavior.
Instruction Tuning
Train a base model on many (instruction, response) pairs to make it follow natural-language commands.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.
LoRA (Low-Rank Adaptation)
Fine-tune by training small low-rank update matrices instead of every weight: same quality, ~1% of the cost.
Pre-training
Self-supervised next-token training on a huge unlabeled corpus, producing the base model.