What is the operational difference between DPO and RLHF?

MCQ·Medium·4.0 · 0·~1 min·Asked atOpenAI Samsung·Relevant atAnthropic Cohere Fireworks Ai Google

Attempt it

**RLHF** (reinforcement learning from human feedback) and **DPO** (direct preference optimization) align a model to human preferences in different ways. Classic RLHF has three stages. First you do supervised fine-tuning (SFT) on good examples. Then you train a separate reward model that scores responses. Finally you run reinforcement learning (usually PPO, a common RL method). The model generates answers, the reward model grades them, and a training loop nudges the model toward higher scores. DPO collapses that. It keeps the SFT step, then skips the reward model and the RL loop. It trains the model directly on pairs of answers (one preferred, one rejected). A single loss raises the probability of the chosen answer and lowers the rejected one. A frozen copy of the SFT model, called the reference, stops the model from drifting too far. A knob called beta controls how tightly it stays anchored. DPO is simpler, cheaper, and more stable. PPO can sometimes reach a higher quality ceiling because it samples fresh responses online, but it is harder to tune. The answer is B. A invents a co-trained reward model, C makes DPO sample online, and D drops the SFT step that DPO keeps.

**DPO** removes two pieces that classic RLHF needs: the separate reward model and the PPO reinforcement-learning loop. That is option B. A invents a co-trained reward model, C wrongly makes DPO sample online, and D wrongly drops SFT, which DPO still needs for its reference. RLHF trains a reward model on preference data. Then it runs PPO, where the policy generates samples online. The reward model scores them. A policy-gradient step pushes toward higher reward. A KL penalty keeps the policy near a reference. That means three stages, four models in memory, and an online sampling loop. DPO uses a closed-form result. The optimal RLHF policy can be written in terms of the reference policy and the reward. You can invert that relationship and express the reward implicitly through the policy. This turns alignment into one supervised-style loss over chosen and rejected pairs. No reward model is trained. No samples are drawn online. The KL to reference constraint does not vanish. It is baked into the loss through the beta term, which controls how far the policy may drift from the frozen reference. DPO is simpler, cheaper, and far more stable. PPO can reach a higher ceiling because online sampling explores responses outside the fixed preference set. Most teams now start with DPO and reach for PPO only when they need that last increment.

TL;DR

DPO drops the reward model and the PPO loop. It optimizes preference pairs directly with a closed-form loss, anchored to a frozen reference by a beta term.

Memory aid

Easy to grasp

Imagine teaching a chef which dish people prefer. The old way hires a food critic to score every plate, then runs a slow trial and error loop where the chef keeps cooking and the critic keeps grading. It works but it is fiddly and expensive. The new way skips the critic entirely. You hand the chef pairs of plates and say plainly: people liked this one more than that one. The chef adjusts directly from those comparisons. A gentle rule keeps the chef from drifting too far from their original style. The goal is the same, with far fewer moving parts. That new way is DPO: learn from preference pairs directly, with no separate scorer and no looping simulation, only a clean supervised-style nudge.

Key concepts

Direct Preference Optimization RLHF / PPO Reward model KL to reference

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Both methods solve the same problem: take a supervised fine-tuned model and align it to human preferences expressed as comparisons between responses. The difference is entirely in the machinery. RLHF reaches the goal through a reward model and a reinforcement-learning loop. DPO reaches the same goal through a single supervised-style loss. It proves that the reward model and the loop are unnecessary in the offline setting.

DPO is not a lighter version of RLHF that swaps in a cheaper reward model. It is a reformulation that removes the reward model analytically and replaces the policy-gradient loop with one offline objective. The operational differences (offline data, no sampling, implicit KL, stability) all follow from the derivation.

The stakes are practical, not academic. A PPO run keeps four models live. It depends on a delicate balance of clip range, value-loss coefficient, and KL coefficient. It can reward-hack a flawed reward model into producing confident garbage. A DPO run looks like a slightly unusual supervised job. For most teams that difference decides whether alignment ships in a week or stalls for a quarter. That is why the open-weight ecosystem swung hard toward DPO after 2023.

What classic RLHF runs

Classic RLHF is a three-stage pipeline.

SFT. Fine-tune the base model on high-quality demonstrations so it follows instructions at all. The result is the policy you will refine and, later, the reference you anchor to.
Reward modelling. Collect human preference pairs and train a separate reward model to assign a scalar score. Training typically uses a Bradley-Terry assumption: the probability of preferring one response is the sigmoid of the reward difference.
PPO. Run reinforcement learning. The policy samples completions, the reward model scores them, and a clipped policy-gradient update maximises expected reward minus a KL penalty to the SFT reference.

The objective being optimised in stage three is reward minus a KL term:

\max_{\pi} \; \mathbb{E}_{x,\,y\sim\pi}\big[r(x,y)\big] - \beta\, D_{KL}\!\big(\pi \,\Vert\, \pi_{\text{ref}}\big)

This works and produced the first aligned chat models. But it keeps up to four models in memory at once: the policy, the reference, the reward model, and the value model. It samples on-policy every step, and it is notoriously sensitive to reward hacking and hyperparameters.

Reward hacking is the failure mode that haunts RLHF. The reward model is only a proxy for human judgement, trained on a finite set of comparisons. The policy is a relentless optimiser pointed straight at that proxy. Given enough steps it finds the gaps: responses that score high on the reward model but that humans would dislike, like sycophancy, padding, or formatting tricks. The explicit KL penalty to the reference is the main guardrail against this. That is one reason the KL term is not optional in either method.

The DPO reparameterisation

Where the KL constraint went

Offline versus on-policy: the real trade-off

The family of variants and how to choose

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	DPO	RLHF (PPO)
Reward model	None (implicit, reparameterised)	Separate trained reward model
Sampling	Offline, precollected pairs	On-policy, samples each step
Loss	Closed-form pairwise log-ratio	Clipped policy gradient on reward
KL to reference	Implicit via beta and frozen reference	Explicit penalty term
Stability and cost	Stable, cheap, simple to run	Fragile, costly, four models in memory
Quality ceiling	Capped by the preference set	Higher with good online reward

Real products, models, and research that use this idea.

Meta's Llama 3 post-training used SFT, rejection sampling and DPO. The paper reports that DPO needed less compute than PPO for large models and performed better.
Hugging Face's TRL library ships DPOTrainer alongside online trainers such as GRPOTrainer and RLOOTrainer. The Zephyr models popularised SFT then DPO as a recipe.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhere exactly does the reward model go in the DPO derivation?

Start from the KL-regularized RLHF objective, write its closed-form optimal policy, then solve for the reward in terms of the policy and reference log-ratio. Substituting into the Bradley-Terry preference model cancels the partition function.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Saying DPO is just RLHF with a smaller reward model. DPO has no reward model and no policy-gradient loop at all; it is a single offline loss over preference pairs.

60 second bullets to scan on the way to the call.

What DPO removes versus what RLHF keeps
The three stages of classic RLHF

Primary sources. Browse if you want the original framing.