Glossary · Training

DPO (Direct Preference Optimization)

Also known as: Direct Preference Optimization

Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.

An alignment method that fine-tunes an LLM directly on preference pairs without training a separate reward model or running PPO. Mathematically equivalent to PPO under specific assumptions, dramatically simpler to implement.

In practice

Increasingly the default replacement for PPO-based RLHF. Knowing why it works (and its caveats vs PPO) is a senior alignment interview staple.

How it compares

vs. RLHF (Reinforcement Learning from Human Feedback) →

RLHF (PPO) trains a reward model then RL-tunes against it; DPO collapses both steps into one supervised loss.

Comparisons that include DPO (Direct Preference Optimization)

DPO vs PPODPO turns RLHF into supervised learning on preference pairs, no reward model needed. PPO still wins on some tasks but is trickier to tune.

Related terms

Fine-tuning

Continue training a pre-trained model on task-specific data to specialize its behavior.

RLHF (Reinforcement Learning from Human Feedback)

Train a reward model from human preference pairs, then RL-fine-tune the LLM against that reward.

LoRA (Low-Rank Adaptation)

Fine-tune by training small low-rank update matrices instead of every weight: same quality, ~1% of the cost.

Knowledge Distillation

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.

Reasoning Model

An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.

Pre-training

Self-supervised next-token training on a huge unlabeled corpus, producing the base model.