Compute the KL shaped reward this PPO rollout actually optimizes

**Shaped reward = 2.0.** The arithmetic: $$\text{shaped\_reward} = r_{\text{RM}} - \beta \cdot \text{KL} = 2.4 - 0.05 \cdot 8.0 = 2.4 - 0.4 = 2.0$$ The **KL term** taxes every nat of drift from the reference policy. The reference policy here is typically the SFT checkpoint that started the run. Beta = 0.05 means each nat of KL costs 0.05 reward; 8 nats of drift cost 0.4. The policy keeps 2.0 of its raw 2.4 reward. **Why this matters.** The policy is gradient-ascending on the shaped reward, not the raw RM score. If beta were 0, the policy would have no incentive to stay anywhere near the reference and would happily drift into RM-exploit regions where the reward model is miscalibrated. The KL tax balances RM reward against drift cost: the policy keeps changes only when the reward gain is worth the drift it paid for. This is the central mechanism that prevents PPO from reward-hacking the RM out of the box.

**Shaped reward is 2.0.** $$\text{shaped\_reward} = r_{\text{RM}} - \beta \cdot \text{KL}(\pi \,\Vert\, \pi_{\text{ref}}) = 2.4 - 0.05 \cdot 8.0 = 2.0$$ The policy gradient is computed against this shaped value, not the raw RM score. Each nat of drift from the reference policy costs `beta` units of reward, so the policy is incentivized to keep drift to the level where the marginal RM gain still exceeds the marginal KL cost. **Mechanism.** The KL penalty is what keeps PPO honest. Without it, the policy would maximize the raw RM score, and the RM has many exploitable regions outside its training distribution. The penalty creates an equilibrium where the policy only spends drift on changes that produce real (RM-validated) reward gain. In practice the KL is computed either at the sequence level (one number per rollout, as here) or per-token (a coefficient on each step's reward). Beta is usually swept across a small range like 0.01 to 0.1; too low and the policy reward-hacks, too high and the policy barely moves. The right beta places the trained policy at the gold-signal peak of the reward-overoptimization curve. For this specific rollout: 8 nats is typical to high mid-run drift; 0.4 reward tax is modest; the resulting shaped reward of 2.0 is what flows into PPO's advantage computation and ultimately into the gradient.