GRPO with group size 4 on a verifiable math task: all four sampled solutions pass the checker, so the group rewards are [1.0, 1.0, 1.0, 1.0]. GRPO computes each sample's advantage as (reward - group_mean) / group_std (with an epsilon guarding the division). What advantages does this group produce, and what policy gradient signal results?
All advantages are 0. GRPO's signal is intra-group contrast; a saturated group (all pass or all fail) teaches the policy nothing.
Imagine a teacher grades four students on a quiz to rank them, but they all get 100. The teacher cannot say who improved most because no one stood out. The grading curve has no slope. So no student gets corrective feedback from that quiz. GRPO works the same way: it compares rollouts to each other in the same prompt, and when they all tie, there is nothing to compare. The fix is to give the class a harder quiz next time, or to throw the all-perfect quiz out and only train on the ones where students actually differed.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
GRPO is one of the most important RL algorithms in modern LLM post-training, sitting behind DeepSeek-R1, Kimi K1.5, and a growing share of open-weight reasoning recipes. Its defining feature is the absence of a value network: instead of training a critic to estimate the baseline, GRPO uses the mean of the rollouts in each prompt's group as the baseline directly. This makes the algorithm simpler, cheaper, and easier to scale, at the cost of one specific failure mode that this question is built around.
When every rollout in a group earns the same reward, the group mean equals every individual reward, the advantage for every sample is zero, and the prompt teaches the policy nothing. This is structural, not a bug.
This deep dive walks through the exact arithmetic, why GRPO's design produces this failure mode, why it is so common in RLVR training, and how production teams mitigate it.
The arithmetic — why every advantage is zero
GRPO's advantage formula is:
where r_i is the reward for rollout i, r-bar is the mean reward over the group, sigma is the group standard deviation, and epsilon is a small constant (typically 1e-8) that guards the division when sigma is small.
With rewards [1.0, 1.0, 1.0, 1.0], the mean is 1.0 and the standard deviation is 0. For each rollout, the numerator r_i - r-bar is 1.0 - 1.0 = 0. The epsilon in the denominator makes the division well-defined (0 / (0 + epsilon) = 0), but it does not create signal from a zero numerator.
The PPO-style clipped surrogate objective then multiplies the log-probability ratio by the advantage. Multiplying by zero gives zero. Zero gradient flows from this prompt's group into the policy update for that batch. The only term that can still nudge the policy from this prompt is the KL penalty against the reference policy, which pulls the policy weights toward the reference rather than toward the rollouts — useful for stability, not for learning the task.
Run the GRPO math. Advantage in GRPO is where and are the within-group mean and std of rewards across G rollouts of the same prompt. If every , then , , and the formula is $0/0\sigma_g$ to a small epsilon, so the numerator zeros out: every advantage becomes 0, every policy gradient term becomes 0, and the update for that prompt contributes nothing to the gradient. Multiply across many flat-group prompts and the effective batch size shrinks.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-R1's GRPO recipe on math training filters all-pass and all-fail groups and schedules problem difficulty to keep most groups mixed
- Kimi K1.5 and similar reasoning-RL pipelines instrument saturated-group fraction as a primary training-health signal
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does raising group size from 4 to 16 change the saturation rate?
Model each rollout as Bernoulli with policy pass-rate p; probability of all-pass is p^G, of all-fail is (1-p)^G; both shrink fast as G grows.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a high mean reward means a high learning signal; in GRPO the magnitude does not matter, only the spread inside the group.
60 second bullets to scan on the way to the call.
The GRPO advantage formula and its inputs
Why intra-group contrast is the only signal GRPO has
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.