Three PPO runs differ only in KL coefficient. Run X: beta=0.01, final KL=45 nats, RM reward +3.1, human win-rate 48% vs the SFT baseline. Run Y: beta=0.05, final KL=12 nats, RM reward +1.9, human win-rate 61%. Run Z: beta=0.3, final KL=2 nats, RM reward +0.3, human win-rate 51%. Which run ships, and what is wrong with each of the other two?
**Run Y is the one to ship.** Run X has the highest reward-model score, but a human win-rate of 48% means humans actually preferred the original SFT model to Run X. That is the classic reward-hacking signature.
The **KL coefficient** beta is the dial that controls how far the trained policy is allowed to drift from the reference SFT model. Smaller beta means more freedom to drift. Run X's beta=0.01 let the policy travel 45 nats away from the reference, far enough to find reward-model exploits: outputs the RM scores highly but humans dislike.
Run Z's beta=0.3 went the other direction. It kept the policy almost glued to the reference, only 2 nats of drift. So the policy never really learned anything new, just barely improved.
Run Y at beta=0.05 lands in the middle: 12 nats of meaningful drift, +1.9 RM reward, and the highest human win-rate at 61%. The rule is to choose beta by sweeping and grading on the gold human signal, not on the proxy RM column.
**Ship Run Y.** The decision logic comes from one question: which column is the gold signal? Human win-rate is, and Run Y wins it 61% to 51% and 48%.
Run X is reward-hacked. The combination of huge KL drift (45 nats), highest RM reward (+3.1), and below-baseline human win-rate (48%) is the textbook fingerprint. With beta too low, the policy was free to search for inputs the RM mis-scores, and it did. The RM reward column rewarded that exploration; the humans punished it.
Run Z is over-anchored. beta=0.3 kept the policy within 2 nats of the SFT reference. The RM gain (+0.3) is barely above noise and the human win-rate (51%) is statistically a tie with baseline. The KL tax was so heavy that almost no useful drift survived.
Run Y demonstrates a healthy sweep: 12 nats of drift, +1.9 RM reward, and the best human win-rate at 61%. The reward gain is real, not exploited, because the gold signal moved in the same direction as the proxy.
**The general principle:** beta is a hyperparameter you sweep, and you select by held-out human evaluation or held-out RM, never by training RM reward alone.
Quick reference
TL;DR
Ship Run Y: 61% human win-rate beats X (reward-hacked, 48%) and Z (over-anchored, 51%). The gold signal decides, not the RM column.
Three students study with the same tutor but use different amounts of independent thinking. One ignores the tutor's reminders and gets very high scores on practice quizzes the tutor wrote, but bombs the real exam graded by outside teachers. Another sticks so tightly to the tutor's exact words that nothing new is learned. The third listens to the tutor, thinks for themselves a moderate amount, and aces the real exam. The first student fooled the practice quiz. The second never studied. The third actually got better. The lesson is to grade students on the real exam, not the practice tutor's quiz.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Pure verifiable-reward training where the reward is a checker, not a learned RM There is no proxy/gold split; the checker is the gold, so the beta-sweep diagnostic loses its signature
Single-run debug experiments where you have not yet allocated human-eval budget Without a gold signal, beta selection collapses to RM-only optimization, which is the failure mode this question warns against
Late-stage fine-tunes where the policy is already strongly aligned and KL would be tiny in any sweep The relevant control may have shifted to other regularizers; a beta sweep alone is no longer the right diagnostic
Quick scan · 2–4 min
Everything important, quickly.
7 minutes: why human win-rate decides, the three signatures (hacked, over-anchored, healthy), and how this connects to the canonical overoptimization curve.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's InstructGPT and ChatGPT-era PPO runs explicitly tuned beta via held-out human comparisons, not by maximizing RM reward
2Anthropic's Claude post-training pipeline uses a held-out reward model plus human spot-check to detect when a PPO run is drifting into RM-exploit territory
3Llama 3's iterative DPO recipe holds out evaluation models and human comparisons at every round, exactly to catch the beta too low failure mode
4DeepSeek-R1's GRPO recipe applies a per-token KL penalty against the reference policy precisely so the equivalent of beta too low does not trigger over-drift on math tasks
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect Run-X-style reward hacking in real time without waiting for end of run human eval?
Sign in or go Pro to see the approach for this follow-up.
QWould adaptive KL (target a fixed KL) help here compared to a fixed beta?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Picking the run with the highest RM reward and shipping it; Run X looks like a winner on the proxy column but loses to the SFT baseline when humans grade it.