Compute the KL shaped reward this PPO rollout optimizes

**Shaped reward = 2.0.** The arithmetic: $$\text{shaped\_reward} = r_{\text{RM}} - \beta \cdot \text{KL} = 2.4 - 0.05 \cdot 8.0 = 2.4 - 0.4 = 2.0$$ The **KL term** (a measure of drift) taxes every nat (natural-log unit of KL) of drift from the reference policy. The reference policy here is typically the supervised fine-tuned model that started the run. Beta = 0.05 means each nat of KL costs 0.05 reward; 8 nats of drift cost 0.4. The policy keeps 2.0 of its raw 2.4 reward. **In practice.** The policy is trained to raise the shaped reward, not the raw reward model (RM) score. If beta were 0, the policy would have no reason to stay anywhere near the reference. It would drift into RM-exploit regions where the reward model gives wrong scores. The KL penalty balances RM reward against drift cost: the policy keeps changes only when the reward gain is worth the drift it paid for. This is the central mechanism that prevents the training from reward-hacking the RM right away.

**Shaped reward is 2.0.** $$\text{shaped\_reward} = r_{\text{RM}} - \beta \cdot \text{KL}(\pi \,\Vert\, \pi_{\text{ref}}) = 2.4 - 0.05 \cdot 8.0 = 2.0$$ The policy gradient is computed against this shaped value, not the raw RM score. Each nat of drift from the reference policy costs `beta` units of reward. So the policy has a reason to keep drift to the level where the marginal RM gain still exceeds the marginal KL cost. **Mechanism.** The KL penalty is what keeps PPO honest. Without it, the policy would maximize the raw RM score, and the RM has many exploitable regions outside its training distribution. The penalty creates an equilibrium where the policy only spends drift on changes that produce real (RM-validated) reward gain. In practice the KL is computed either at the sequence level (one number per rollout, as here) or per token (a coefficient on each step's reward). Beta is usually swept across a small range like 0.01 to 0.1. Too low, and the policy reward-hacks. Too high, and the policy barely moves. The right beta places the trained policy at the gold-signal peak of the reward-overoptimization curve. For this rollout, 8 nats is typical to high mid-run drift, and a 0.4 reward penalty is modest. The resulting shaped reward of 2.0 is what flows into PPO's advantage computation and then into the gradient.