Why is a KL penalty against the reference policy necessary in PPO based RLHF?

Short answer·Medium·4.0 · 0·~3 min·Asked atBaidu Copy Ai

Attempt it

Free · 2 AI evals / day

TL;DR

The reward model is an imperfect proxy, so a KL penalty keeps the PPO policy near the reference: this blocks reward hacking, keeps fluency and stops collapse onto one answer.

Memory aid

Easy to grasp

Picture a student whose essays are graded by a computer program. The program was built from a few hundred essays that teachers marked. It mostly works, but it has blind spots: it gives extra points for long essays and certain fancy words. If the student chases only the program's score, the essays fill with long, strange, repeated words that score well and read badly. So the teacher adds a rule: your essay can improve, but it must still sound like your own writing from last term. Big changes cost points. That rule is the KL penalty. The student still gets better, but stays where the grading program gives honest scores.

Key concepts

RLHF

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

PPO-based RLHF fine-tunes a policy to get high scores from a learned reward model. The KL penalty adds a cost for moving away from a frozen reference policy, usually the SFT model. It looks like a small regularizer, but RLHF fails without it.

The reason is that the reward model is a proxy. It was trained on a limited set of outputs and is accurate only near them. A strong optimizer finds outputs outside that region that score high and read badly. The KL penalty keeps the policy in the region where the reward means something. It also keeps outputs fluent and varied, and beta controls how far the policy may move.

The objective: reward minus distance from the reference

RLHF with PPO does not maximize the reward model score on its own. It maximizes the score minus a penalty for moving away from a frozen reference policy, usually the SFT model:

J(\pi)=\mathbb{E}_{x\sim D,\,y\sim\pi}\big[r_\phi(x,y)\big]-\beta\,D_{KL}\big(\pi(\cdot\mid x)\,\Vert\,\pi_{ref}(\cdot\mid x)\big)

In practice the penalty goes into the reward that PPO sees. For each sampled response, the shaped reward is the reward model score minus β times the log ratio log π(y|x) − log π_ref(y|x). InstructGPT applies this as a per-token KL penalty from the SFT model at every token, with β = 0.02. The log ratio is summed over the tokens of the response, so its expected value is the KL divergence for that prompt.

The two terms pull in different directions. The reward term moves probability toward responses the reward model scores highly. The KL term charges a cost wherever the policy gives a token more probability than the reference does. β sets the exchange rate between them: how much reward the policy must gain to justify one more nat of divergence.

Most preference methods share this objective. PPO optimizes it with sampled rollouts. DPO optimizes the same KL-regularized objective through a closed-form reparameterization. So its β plays the same role, even though no reward model or explicit KL term appears in its loss.

Why the reward model needs an anchor

The optimal policy shows what beta does

What happens without the penalty

Choosing beta, and what the penalty cannot do

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

InstructGPT added a per-token KL penalty from the SFT model with coefficient 0.02. Human Likert scores were poor at both 0 and 2.0.
In Fine-Tuning Language Models from Human Preferences, a policy trained on a mock sentiment reward with no KL penalty scored about 99.97% positive while writing repeated-word gibberish.

What an interviewer would ask next. Try answering before peeking at the approach.

QThe KL-regularized objective has the optimum π* ∝ π_ref · exp(r/β). What does this imply as β goes to zero, and as β grows large?

As β shrinks, the exponent dominates and all probability moves onto the highest-reward response, including any reward-model error. As β grows, exp(r/β) goes to 1 and the policy stays at the reference.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating the PPO clip as enough regularization. The clip limits each step against the old policy; only the KL term stops slow drift away from the frozen reference model.

60 second bullets to scan on the way to the call.

KL-regularized RLHF objective and the role of beta
Closed-form optimal policy under a KL constraint

Primary sources. Browse if you want the original framing.