25 side by side comparisons of the concepts interviewers ask about most. Understand the tradeoffs, know when to use what, and never confuse the two again.
DDP replicates the model per GPU; FSDP and ZeRO shard it, freeing memory so bigger models fit. ZeRO is a spec (stages 1-3); FSDP is a fully-shardable PyTorch implementation of the same idea.
DPO turns RLHF into supervised learning on preference pairs, no reward model needed. PPO still wins on some tasks but is trickier to tune.
In-context learning adapts at inference by putting examples in the prompt; fine-tuning adapts by updating weights. Reach for prompting first; fine-tune when you cannot get there.
LoRA reaches 90-99% of full fine-tuning quality for a fraction of the compute and storage; use full fine-tuning only when that last margin is genuinely worth it.