Fill in the GRPO group relative advantage normalization step
GRPO advantage subtracts the group mean reward and divides by group standard deviation: A_i = (R_i - mean) / std.
When the model tries five answers to the same question, you grade each answer. GRPO asks how much better or worse each answer is compared to the average of the group, not compared to some separate judge model. Subtract the average score, then divide by how spread out the scores were so big and small groups behave similarly.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The GRPO advantage formula is the critic replacement mechanism. It is the core of group-relative policy optimization: center rewards at the group mean, scale by group spread.
If you get this wrong, every downstream claim about DeepSeek-style RLVR training is unsound.
The template and accepted fills
Given rewards for completion in a group sampled from the same prompt:
First blank: mean of group rewards, also written group mean, μ, mu.
Second blank: standard deviation of group rewards: std, σ, sigma, group std.
Implementations add ε to σ_G to keep the division stable when variance is tiny.
On a whiteboard: write K samples and rewards R_i, subtract the mean, then divide by the std.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-R1 GRPO implementation normalizes verifiable math rewards within each prompt group
- DAPO dynamic sampling ensures reward diversity so group std stays informative
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy divide by standard deviation instead of only centering at the mean?
It stabilizes scale across groups with different reward spreads. Gradient magnitudes stay comparable when one prompt's rewards are tight and another's are wide.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Filling the blanks with global batch statistics or PPO value estimates instead of within-group mean and standard deviation.
60 second bullets to scan on the way to the call.
Write A_i = (R_i - mean) / std for the same prompt group
Contrast with PPO critic baseline
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.