Why has GRPO eaten PPO's lunch in 2025-2026 reasoning post-training?

Short answer·Medium·4.0 · 0·~3 min·Relevant atAnthropic Google Meta OpenAI

Attempt it

GRPO (Group Relative Policy Optimization) has largely replaced PPO in 2025-2026 reasoning post-training recipes (DeepSeek-R1, Qwen2.5/3-Math, several open reasoning models). Explain the single biggest structural change GRPO makes versus PPO, the concrete memory and infrastructure win that follows, and one trade-off you give up by dropping that component.

Free · 2 AI evals / day

**PPO** is the classic RL algorithm used in RLHF stages of model training. It updates the policy (the model being trained) by computing an advantage for each generated response, then taking a small clipped policy step in the direction that increases reward. To make the advantage stable, PPO needs a baseline that estimates the expected reward at each state. It uses a learned critic network for this. The critic is roughly the same size as the policy, gets its own optimizer states, and has to be kept in sync during distributed training. At a 70 billion parameter scale, this is a serious cost. GRPO removes the critic entirely. Instead of learning a value function, GRPO generates a group of N completions for each prompt, takes their mean reward as the baseline, and defines the advantage as the per-rollout reward minus that mean, normalised by the group standard deviation. The group becomes its own baseline. The payoff is large. No critic means half the training-time model memory and one fewer model to coordinate across GPUs. DeepSeek-R1 and the Qwen reasoning models both used GRPO and showed it scales cleanly. The trade-off is variance. A learned critic is a smooth low-variance baseline. A small-group mean is noisier, so GRPO works best with larger group sizes, typically 8 to 16 rollouts per prompt.

**The single structural change** is that GRPO removes the critic network. PPO needs a learned value function V(s) so the advantage estimate A equals R minus V(s) has low variance. The critic is a second model, roughly the size of the policy, with its own optimizer states and gradient flow. GRPO replaces this learned baseline with a group-relative one: for each prompt, sample N rollouts, take their mean reward as the baseline, and define the advantage as the per-rollout reward minus mean, often divided by the group standard deviation. **The memory and infra win follows directly.** Dropping the critic cuts training-time model VRAM roughly in half on the model side, since the critic is policy-sized, and removes its optimizer states too. The infrastructure simplification is just as important. One fewer model means one fewer model to shard, one fewer set of collectives to coordinate, and one fewer training loop to keep in lockstep. At 70 billion parameters this is the difference between a clean recipe and a fragile one. **The trade-off** is variance. A learned critic produces a smooth, low-variance baseline that captures expected reward across many similar states. A sample-mean baseline is noisier, especially with small group sizes. The fix is to push the group size up, typically to 8 or 16 rollouts per prompt, so the within-group variance averages out enough for stable updates. **Concrete adoption.** DeepSeek-R1 used GRPO as its core RL stage, the Qwen 2.5 and 3 Math reasoning recipes rely on it heavily, and most 2026 open reasoning stacks now default to it.

Quick reference

TL;DR

GRPO drops PPO's critic network and uses the mean reward across N rollouts per prompt as the baseline. Big memory and infra savings at scale, the trade is higher variance unless N is large enough.

Memory aid

Easy to grasp

Picture grading a class of essays without a fixed answer key. With a key, every essay gets compared to one ideal and you mark each one above or below it. With no key, you instead read several essays on the same topic, take the class average, and grade each essay as better or worse than the rest of the batch. The class becomes its own benchmark. The first method needs a teacher who has spent ages writing keys; the second method needs only a stack of essays per topic. It is cheaper and faster, but the grades wobble if the batch is small and the topic is unusually easy or hard. GRPO is the second method applied to model rollouts.

Key concepts

GRPO RLHF Reasoning models PPO

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: critic removal as the defining change + group-mean baseline formula + normalisation by group std + memory and infra accounting + variance trade-off at small N + adoption in DeepSeek-R1 and Qwen + when PPO is still preferred.

Real-world examples

Real products, models, and research that use this idea.

DeepSeek-R1 used GRPO as its core RL stage in 2025, and the published recipe became the reference implementation for open reasoning models.
Qwen 2.5 Math and Qwen 3 Math reasoning families adopted GRPO and showed it scales cleanly to 70B-class policies with group sizes of 16.
Open-R1, the community open-source reproduction of DeepSeek-R1, defaults to GRPO via the verl training stack and documents the memory savings explicitly.
Hugging Face TRL ships a GRPOTrainer in 2026, slotting next to PPOTrainer and DPOTrainer as a standard option for RL post-training.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does GRPO normalise the advantage by the group standard deviation rather than just subtracting the mean?

QWhat happens to GRPO when the reward signal is very sparse, say only one rollout in 16 gets non-zero reward?

QHow would you decide between GRPO and DPO for a reasoning task?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Calling GRPO a small tweak to PPO. The defining change is removing the critic entirely, which is what unlocks the memory and infra savings. Without that, you do not have GRPO.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.