Why has GRPO eaten PPO's lunch in 2025-2026 reasoning post-training?

**PPO** is the classic RL algorithm used in RLHF stages of model training. It updates the policy (the model being trained) by computing an advantage for each generated response, then taking a small clipped policy step in the direction that increases reward. To make the advantage stable, PPO needs a baseline that estimates the expected reward at each state. It uses a learned critic network for this. The critic is roughly the same size as the policy, gets its own optimizer states, and has to be kept in sync during distributed training. At a 70 billion parameter scale, this is a serious cost. GRPO removes the critic entirely. Instead of learning a value function, GRPO generates a group of N completions for each prompt, takes their mean reward as the baseline, and defines the advantage as the per-rollout reward minus that mean, normalised by the group standard deviation. The group becomes its own baseline. The payoff is large. No critic means half the training-time model memory and one fewer model to coordinate across GPUs. DeepSeek-R1 and the Qwen reasoning models both used GRPO and showed it scales cleanly. The trade-off is variance. A learned critic is a smooth low-variance baseline. A small-group mean is noisier, so GRPO works best with larger group sizes, typically 8 to 16 rollouts per prompt.

**The single structural change** is that GRPO removes the critic network. PPO needs a learned value function V(s) so the advantage estimate A equals R minus V(s) has low variance. The critic is a second model, roughly the size of the policy, with its own optimizer states and gradient flow. GRPO replaces this learned baseline with a group-relative one: for each prompt, sample N rollouts, take their mean reward as the baseline, and define the advantage as the per-rollout reward minus mean, often divided by the group standard deviation. **The memory and infra win follows directly.** Dropping the critic cuts training-time model VRAM roughly in half on the model side, since the critic is policy-sized, and removes its optimizer states too. The infrastructure simplification is just as important. One fewer model means one fewer model to shard, one fewer set of collectives to coordinate, and one fewer training loop to keep in lockstep. At 70 billion parameters this is the difference between a clean recipe and a fragile one. **The trade-off** is variance. A learned critic produces a smooth, low-variance baseline that captures expected reward across many similar states. A sample-mean baseline is noisier, especially with small group sizes. The fix is to push the group size up, typically to 8 or 16 rollouts per prompt, so the within-group variance averages out enough for stable updates. **Concrete adoption.** DeepSeek-R1 used GRPO as its core RL stage, the Qwen 2.5 and 3 Math reasoning recipes rely on it heavily, and most 2026 open reasoning stacks now default to it.