GRPO with group size 4 on a verifiable math task: all four sampled solutions pass the checker, so the group rewards are [1.0, 1.0, 1.0, 1.0]. GRPO computes each sample's advantage as (reward - group_mean) / group_std (with an epsilon guarding the division). What advantages does this group produce, and what policy gradient signal results?
All advantages are 0. GRPO's signal is intra-group contrast; a saturated group (all pass or all fail) teaches the policy nothing.
Imagine a teacher grades four students on a quiz to rank them, but they all get 100. The teacher cannot say who improved most because no one stood out. The grading curve has no slope. So no student gets corrective feedback from that quiz. GRPO works the same way. It compares rollouts to each other in the same prompt, and when they all tie, there is nothing to compare. The fix is to give the class a harder quiz next time. Or throw the all-perfect quiz out and only train on the ones where students differed.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
GRPO, introduced in DeepSeekMath, is a widely used RL algorithm in LLM post-training and the one behind DeepSeek-R1 and many open-weight reasoning recipes. Its defining feature is the absence of a value network. Instead of training a critic to estimate the baseline, GRPO uses the mean reward of the rollouts in each prompt's group as the baseline directly. This makes the algorithm simpler, cheaper, and easier to scale, at the cost of one specific failure mode.
When every rollout in a group earns the same reward, the group mean equals every individual reward. The advantage for every sample is zero, and the prompt teaches the policy nothing. This is structural, not a bug.
The arithmetic: why every advantage is zero
GRPO's advantage formula is:
where r_i is the reward for rollout i, r-bar is the mean reward over the group, and sigma is the group standard deviation. Epsilon is a small constant (TRL uses 1e-4) that guards the division when sigma is small.
With rewards [1.0, 1.0, 1.0, 1.0], the mean is 1.0 and the standard deviation is 0. For each rollout, the numerator r_i - r-bar is 1.0 - 1.0 = 0. The epsilon in the denominator makes the division well-defined (0 / (0 + epsilon) = 0), but it does not create signal from a zero numerator.
The PPO-style clipped surrogate objective then multiplies the log-probability ratio by the advantage. Multiplying by zero gives zero. Zero gradient flows from this prompt's group into the policy update for that batch. The only term that can still nudge the policy from this prompt is the KL penalty against the reference policy. It pulls the policy weights toward the reference rather than toward the rollouts. That is useful for stability, not for learning the task.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeekMath introduced GRPO with 64 samples per question; DeepSeek-R1-Zero used 16 samples per question with rule-based accuracy and format rewards, the binary setting where flat groups are common
- DAPO's dynamic sampling over-samples prompts and filters out those whose group accuracy is exactly 1 or 0, so every kept group carries a gradient
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does raising group size from 4 to 16 change the saturation rate?
Model each rollout as Bernoulli with policy pass-rate p; probability of all-pass is p^G, of all-fail is (1-p)^G; both shrink fast as G grows.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a high mean reward means a high learning signal; in GRPO the magnitude does not matter, only the spread inside the group.
60 second bullets to scan on the way to the call.
The GRPO advantage formula and its inputs
Why intra-group contrast is the only signal GRPO has
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.