Define GRPO and why it eliminates the critic/value network
GRPO scores sampled outputs relative to their group, using normalized advantages, so it can optimize policies without a separate critic/value network.
Imagine grading students in small groups where you compare each answer to the group's average quality. You can tell which answers are better or worse without building a separate teacher model that predicts scores for every case. GRPO does this for model outputs: relative ranking inside the sampled group gives the learning signal. There is one catch. If everyone in a group gets the question right, or everyone gets it wrong, the comparison says nothing, and that question teaches nothing this round.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Group Relative Policy Optimization (GRPO) is a variant of PPO introduced with DeepSeekMath. It keeps PPO's clipped policy update but throws away the value network, the critic that PPO trains next to the policy.
The critic's only job in PPO is to supply a baseline: an estimate of how much reward a typical response to this prompt earns. GRPO gets that baseline another way. It samples a group of responses to the same prompt and uses their average reward. Once the baseline comes from the group, the critic has nothing left to do.
What the critic does in PPO
A policy-gradient update pushes up the log-probability of an action in proportion to its advantage: how much better the action turned out than expected. Using raw reward instead of advantage still points in the right direction on average, but the variance is huge. If every response to an easy prompt scores high, raw reward pushes all of them up, good and mediocre alike.
PPO subtracts a baseline to fix that. The baseline is the value function V(s), the expected future reward from a given state, and the advantage is roughly reward minus V. In RLHF, PPO learns V with a separate value model, usually initialized from the reward model and about as large as the policy. PPO then computes advantages per token with generalized advantage estimation (GAE).
The critic is expensive and hard to train in this setting. The reward model usually scores only the finished response, so the critic must predict a final score from every partial prefix. A poor critic gives noisy advantages. The critic also adds a fourth large model to memory next to the policy, the frozen reference and the frozen reward model.
None of this changes what the baseline is for. Any estimate of "how good is a typical response to this prompt" reduces variance, as long as it does not depend on the specific response being scored. That opens the door to a cheaper estimate.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeekMath introduced GRPO and sampled 64 outputs per question in its RL stage, with a KL coefficient of 0.04.
- DeepSeek-R1-Zero was trained with GRPO directly on the base model, sampling 16 outputs per question and scoring them with rule-based accuracy and format rewards.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are verifiable rewards a natural fit for GRPO?
Explain that reliable, objective scoring strengthens relative ranking within groups and reduces ambiguity in advantage signals.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Defining GRPO only as "PPO without critic" without explaining group-relative normalization as the replacement signal.
60 second bullets to scan on the way to the call.
Definition of GRPO
How group-relative advantages are computed
Primary sources. Browse if you want the original framing.
- Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Schulman et al., Proximal Policy Optimization Algorithms
- Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective
- Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Same topic, related formats. Practice these next.