Concepts · Training

DPO (Direct Preference Optimization)

Also known as: Direct Preference Optimization

Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.

An alignment method that fine-tunes an LLM directly on preference pairs without training a separate reward model or running PPO. Mathematically equivalent to PPO under specific assumptions, dramatically simpler to implement.

In practice

Increasingly the default replacement for PPO-based RLHF. Knowing why it works (and its caveats vs PPO) is a senior alignment interview staple.

How it compares

vs. RLHF (Reinforcement Learning from Human Feedback) →

RLHF (PPO) trains a reward model then RL-tunes against it; DPO collapses both steps into one supervised loss.

Comparisons that include DPO (Direct Preference Optimization)

DPO vs PPODPO turns RLHF into supervised learning on preference pairs, no reward model needed. PPO still wins on some tasks but is trickier to tune.

Questions that mention this term

DPO vs RLHF: pick the most accurate operational difference

Related terms

Fine-tuning

Continue training a pre-trained model on task-specific data to specialize its behavior.

FSDP (Fully Sharded Data Parallel)

Shard weights+grads+optimizer states across GPUs and gather just-in-time; train huge models with limited per-GPU memory.

GRPO (Group Relative Policy Optimization)

PPO without the critic: advantages are computed by ranking multiple sampled responses against each other.

Instruction Tuning

Train a base model on many (instruction, response) pairs to make it follow natural-language commands.

Knowledge Distillation

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.

LoRA (Low-Rank Adaptation)

Fine-tune by training small low-rank update matrices instead of every weight: same quality, ~1% of the cost.