Estimate the memory gap between PPO RLHF and DPO on a 7B run

Attempt it

You are planning a 7B post-training run. PPO RLHF keeps policy, reference, reward model, and value estimation in memory (~10 GB each in this setup). DPO keeps policy and reference only. Estimate total model memory footprint for each.

TL;DR

PPO RLHF keeps four models in memory (policy, reference, reward, value) and DPO keeps two (policy, reference), so at ~10 GB each PPO needs ~40 GB and DPO ~20 GB.

Memory aid

Sign in to see the mnemonic that makes this stick.

Easy to grasp

Picture two cooking schools. In the PPO school, four people share the kitchen: the student cook, a book of how the student cooked on day one, a judge who tastes every new dish, and a coach who guesses how good each dish will be before the judge tastes it. Each of them needs a work station. In the DPO school there is no live tasting. The student studies a notebook of old dish pairs where someone already marked the better one, and checks the day-one book to see how much they have changed. Only two stations are needed. If every station takes the same floor space, the PPO kitchen needs twice the room.

Key concepts

RLHF

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

PPO RLHF and DPO start from the same supervised (SFT) model and aim at the same goal: a policy that people prefer, kept close to where it started. They reach it in different ways, and the difference shows up first in GPU memory. PPO runs an online reinforcement learning loop that needs four networks at once. DPO turns preference learning into a classification loss on fixed pairs and needs two.

In this setup each 7B copy takes about 10 GB, so PPO needs about 4 × 10 = 40 GB and DPO about 2 × 10 = 20 GB. The 2× ratio holds in real runs. The absolute numbers do not, because a copy that trains costs far more memory than a copy that only runs forward passes.

Why PPO holds four networks

A PPO step has two phases. In the rollout phase, the policy generates answers to a batch of prompts. Then three other networks process those answers:

The reward model reads each prompt and answer and returns one scalar score.
The reference model, a frozen copy of the starting SFT model, gives a log-probability for every generated token. The gap between the policy and reference log-probabilities becomes a per-token KL penalty.
The value model, also called the critic, predicts at each token the reward the policy can still expect from that point on. PPO combines these predictions with generalized advantage estimation (GAE) to get advantages: how much better each token choice did than expected.

The reward that PPO maximizes combines the first two signals:

R(x, y) = r_\phi(x, y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}

In the update phase, the policy and the value model both take gradient steps. The policy follows the clipped PPO objective, and the value model regresses toward the observed returns. So PPO has two networks that train (policy and value) and two that stay frozen (reference and reward).

None of the four is optional in standard PPO. Without the reward model there is no score for new samples. Without the reference there is no KL anchor, and the policy drifts toward reward hacking. Without the value model there is no baseline, and the policy gradient becomes very noisy. The value model is often initialized from the reward model or the SFT model, so it is usually as large as one of them.

Why DPO needs only two

From copy count to gigabytes

Memory the copy count leaves out

Ways to cut the gap

Sign in to unlock the full deep dive.

Situations where this technique stops working.

Sign in to see when this approach fails.

2–4 min · Everything important, quickly.

Sign in to see the quick scan of the deep dive.

Concern	PPO RLHF	DPO
Core setup	Online RL loop with more components	Offline preference objective
Memory footprint	Usually higher	Usually lower
Iteration speed	Slower to tune	Faster to iterate
Best fit	Exploration-heavy regimes	Strong offline preference data

Real products, models, and research that use this idea.

InstructGPT ran PPO with a 6B reward model and a value function initialized from that reward model, even for its 175B policy, to save compute.
Hugging Face TRL's PPOTrainer, removed in TRL 1.13, took separate model, ref_model, reward_model and value_model arguments, while DPOTrainer takes a model and an optional ref_model.

Sign in to see more production examples.

What an interviewer would ask next. Try answering before peeking at the approach.

QRedo the estimate for a full fine-tune of a 7B policy with mixed-precision Adam.

A

Use about 16 bytes per parameter for each model that trains and 2 bytes for each frozen bf16 model, then decide which PPO models train.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Forgetting that PPO's value model is a full network, or giving every copy the same size when the copies that train also carry gradients and optimizer state.

Sign in to see all red flags and common mistakes.

60 second bullets to scan on the way to the call.

Role of each of the four networks in a PPO step
Why DPO needs no reward model and no value model

Sign in to unlock the revision sheet.