Why did DeepSeek style reasoning models favor GRPO over classical PPO RLHF?

Short answer·Hard·4.0 · 0·~3 min·Asked atCitadel Shield Ai Sourcegraph

Attempt it

Free · 2 AI evals / day

TL;DR

DeepSeek-style reasoning models favored GRPO because group-relative, verifiable-reward training reduced PPO critic overhead while preserving strong learning signals on objective tasks.

Memory aid

Easy to grasp

Imagine a math class where a student solves the same problem eight times and an answer key marks each try right or wrong. To know which tries deserve praise, you do not need a second teacher who predicts how well the student will do. You compare each try with the student's own average on that problem. A right answer on a problem the student usually gets wrong earns a lot of praise. A wrong answer on an easy problem earns a lot of blame. PPO keeps that second teacher (the critic), and it is about as big as the student. GRPO sends the critic home and uses the group average instead. With an answer key that cannot be fooled, that is cheaper and works well.

Key concepts

RLHF

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

GRPO (Group Relative Policy Optimization) is PPO with one network removed. PPO for language models trains a policy and a value model (the critic) and keeps a frozen reference model and a frozen reward model in memory. GRPO keeps PPO's clipped update but replaces the critic with a simple statistic: the average reward of several answers to the same prompt.

DeepSeek introduced GRPO in DeepSeekMath and then used it for DeepSeek-R1. Two reasons drove the choice. The critic is expensive and hard to train when only the last token gets a reward. And reasoning tasks have rewards that a program can check. Once the reward is a reliable right-or-wrong check, the group of answers gives a good baseline for free.

What the critic does in PPO

A policy gradient pushes up the log-probability of tokens that led to a better than expected result. "Better than expected" needs a baseline. In PPO the baseline comes from a value model V that predicts, at each token, the reward the answer will end with. The advantage at token t is roughly the reward that followed minus V at that token, usually smoothed with generalized advantage estimation (GAE).

In RLHF this design has two costs. First, the value model is usually a network about the size of the policy, often initialized from the reward model. It needs its own weights, gradients and optimizer state. So a PPO run holds four large models: the policy and value model (both training), and the reference and reward models (both frozen).

Second, the critic has a hard learning problem. The reward model scores only the finished answer, so the reward arrives at the last token. The critic must still predict a sensible value at every earlier token of a long chain of thought. DeepSeekMath names this directly: with only the last token scored, a value function that is accurate at each token is hard to train. A poor critic gives noisy advantages, and noisy advantages give noisy updates.

The group baseline that replaces the critic

Why verifiable rewards made GRPO the natural fit

What GRPO costs

When PPO with a critic still makes sense

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

DeepSeekMath introduced GRPO, sampling 64 outputs per question with a KL coefficient of 0.04 and a learned reward model.
DeepSeek-R1-Zero ran GRPO directly on the base model with 16 outputs per question and rule-based accuracy and format rewards, without supervised fine-tuning first.

What an interviewer would ask next. Try answering before peeking at the approach.

QA GRPO run on math prompts plateaus, and logs show that 70% of groups have every answer correct or every answer wrong. What is happening, and what would you change?

Those groups have zero standard deviation and zero advantage, so they add no gradient. Filter or over-sample prompts so each batch has mixed groups (DAPO's dynamic sampling), or retune prompt difficulty.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Saying GRPO removes the reward model. It removes the value model (critic); a reward still scores every answer, from a learned model in DeepSeekMath and from rule-based checks in DeepSeek-R1.

60 second bullets to scan on the way to the call.

What PPO's value model does and how big it is
How GRPO computes the group-relative advantage

Primary sources. Browse if you want the original framing.