How GRPO computes advantages without a critic model.

Standard PPO for LLMs trains two networks: a policy that generates text and a critic that predicts how good a partial completion will be. That critic costs memory and can be unstable on long reasoning chains. **GRPO** (Group Relative Policy Optimization) drops the critic. For each prompt it samples a group of completions — often 8 to 64 — scores each with a task reward (math checker, unit tests), then normalizes rewards within that group only. Each completion's advantage is its z-score relative to siblings on the same prompt. Better-than-average samples get positive signal; worse get negative. The policy gradient update uses those relative advantages, cutting VRAM versus PPO while still learning from comparative feedback.

GRPO replaces PPO's value baseline with **group-relative advantage estimation**. For each training prompt you sample K completions from the current policy, score each with a verifiable reward, then normalize within the group: $$A_i = \frac{r_i - \mu_G}{\sigma_G + \epsilon}$$ where $\mu_G$ and $\sigma_G$ are the mean and standard deviation of rewards across the K siblings. That z-scored standing becomes the advantage for the policy gradient — no separate critic network is trained. The win is memory: you avoid storing and backpropping through a value head on long CoT trajectories. DeepSeek-R1 and similar open recipes lean on GRPO for RLVR because math and code rewards are cheap to compute at scale. Trade-off: you need enough group diversity — if all K samples fail identically, $\sigma_G \approx 0$ and learning stalls.