Glossary · Training

FSDP (Fully Sharded Data Parallel)

Also known as: Fully Sharded Data Parallel, ZeRO-3

Shard weights+grads+optimizer states across GPUs and gather just-in-time; train huge models with limited per-GPU memory.

A data-parallel training strategy that shards model weights, gradients, and optimizer states across GPUs and gathers parameters just-in-time per forward/backward pass. PyTorch's open implementation of the ZeRO-3 algorithm.

In practice

The default for open-source large-model training. Senior training interviews probe FSDP vs DeepSpeed ZeRO and the bandwidth trade-offs.

How it compares

vs. Tensor Parallelism →

FSDP shards data-parallel state (weights, grads, opt); TP shards model compute within layers.

Comparisons that include FSDP (Fully Sharded Data Parallel)

DDP vs FSDP vs ZeRODDP replicates the model per GPU; FSDP and ZeRO shard it, freeing memory so bigger models fit. ZeRO is a spec (stages 1-3); FSDP is a fully-shardable PyTorch implementation of the same idea.

Related terms

Fine-tuning

Continue training a pre-trained model on task-specific data to specialize its behavior.

LoRA (Low-Rank Adaptation)

Fine-tune by training small low-rank update matrices instead of every weight: same quality, ~1% of the cost.

DPO (Direct Preference Optimization)

Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.

Knowledge Distillation

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.

Pre-training

Self-supervised next-token training on a huge unlabeled corpus, producing the base model.

Supervised Fine-Tuning (SFT)

Fine-tune the base model on (prompt, ideal response) pairs; the first post-training step before alignment.