Compare

DPO vs PPO

Direct Preference Optimization vs Proximal Policy Optimization for RLHF

The verdict

DPO turns RLHF into supervised learning on preference pairs, no reward model needed. PPO still wins on some tasks but is trickier to tune.

DPO

Glossary →

Direct Preference Optimization skips the separate reward model. It trains directly on chosen vs rejected pairs using a closed-form loss derived from the RLHF objective. Simpler, cheaper, more stable.

Best for: Simpler alignment with fewer moving parts.

PPO

Glossary →

Proximal Policy Optimization uses a reward model to score generations and updates the policy with a clipped objective plus a KL penalty against the reference model. The classic RLHF recipe.

Best for: Cases where an explicit reward model already exists.

At a glance

DPO vs PPO: dimension-by-dimension comparison
Dimension	DPO	PPO
Reward model	Not needed	Required
Training loop	Supervised on pairs	Rollout + reward + policy update
Stability	High	Sensitive to hyperparameters
Memory	≈ SFT	SFT + reward + critic
Data shape	Static preference pairs	Prompts + rollouts + rewards
Best for	Most preference-tuning today	Explicit reward signal or online RL

Key differences

1DPO removes the separate reward model; PPO requires one
2DPO is more stable and easier to tune; PPO needs KL scheduling, value networks, and reward normalization
3PPO handles online rollouts naturally; DPO is off-policy on a fixed preference set
4DPO's memory footprint is roughly SFT-size; PPO adds critic + reward model
5Recent variants (IPO, cDPO, KTO) extend DPO's idea rather than PPO's

In the interview

What they're really testing

Whether you can explain why DPO exists (PPO is fussy) and what DPO gives up (online exploration).

Say this

PPO is the original RLHF recipe: train a reward model on preferences, then update the policy against it with a clipped objective and a KL to the reference. DPO derives a closed-form loss that reproduces that objective directly from preference pairs, cutting the reward model out. That makes DPO cheaper and much more stable, at the cost of being off-policy and locked to a static preference set. Most teams start with DPO now.

Traps to sidestep

Saying DPO is just SFT
Missing the role of the KL penalty against the reference in both methods
Claiming PPO is dead (still competitive on some tasks and needed for online RL)
Forgetting that DPO variants (IPO, cDPO, KTO) address DPO's overfitting on noisy pairs

How to choose

If you have static preference pairs and want stabilityDPO

If you already have a strong reward modelPPO

If online exploration mattersPPO

If compute and stability are the bottleneckDPO

Default to DPO; reach for PPO when you need online RL or an explicit reward model.

Common misconceptions

Myth: DPO is a completely different objective from PPO.

Reality: DPO derives its loss from the same KL-regularized RLHF objective PPO optimizes. It's the same target, solved differently.

Myth: DPO always beats PPO.

Reality: DPO overfits noisy preference pairs; variants like IPO and cDPO exist because of it. PPO still wins on some benchmarks.

Memory aid

PPO is 'try, get a grade, adjust'; DPO is 'here's the answer key, learn it.'

Can you combine them?

In practice teams typically pick one. But DPO variants (IPO, cDPO, KTO) can be stacked as ablations to find the most robust loss on a given preference set.

Related comparisons

DDPvsFSDPvsZeRO

DDP vs FSDP vs ZeRO

Three ways to shard distributed LLM training across GPUs

DistillationvsQuantization

Distillation vs Quantization

Two orthogonal ways to shrink a large language model

In-context learningvsFine-tuning

In-context learning vs Fine-tuning

Adapt in the prompt, or adapt in the weights

LLM EvaluationvsRLHF (Alignment Training)

LLM Evaluation vs RLHF

Measuring model quality vs training models to be better

LoRA (Low-Rank Adaptation)vsFull Fine-tuning

LoRA vs Full Fine-tuning

Parameter-efficient adaptation vs updating all model weights

RAGvsFine-tuning

RAG vs Fine-tuning

Two approaches to giving LLMs domain specific knowledge