GRPO and RLOO drop PPO's learned value network and use group or leave one out baselines instead. Defend the design: what did the critic buy you in PPO, what replaces it in the value free methods, and in which regimes does critic free RL struggle?
The PPO critic gave per-token advantages via GAE for fine credit assignment but cost a second large model and was hard to fit on text prefixes.
PPO had a second helper model that guessed how good the situation looked at every word, so the agent could tell which specific words mattered most. That helper was expensive and itself hard to train. GRPO and RLOO say: forget the helper, just generate a handful of full attempts for the same prompt and use their average score as the baseline. Each attempt's advantage is how much it beat the average. Cheaper, simpler, and works great when the reward only arrives at the end of the sequence (like a math answer being right or wrong). The catch is you lose the fine-grained 'which word mattered' information; for long multi-step tasks where only some steps caused the outcome, the critic-less methods cannot tell which steps to reinforce.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The shift from PPO with a learned value head to value-free methods (GRPO, RLOO) is one of the consequential structural changes in LLM RL between 2023 and 2025. The motivation is not algorithmic taste; it is the recognition that the learned value head was the most expensive and most unstable part of PPO for the reward structure that dominates modern LLM training. Replacing it with an empirical baseline computed from grouped sampling halves the memory and compute, eliminates the value-fitting instability, and fits naturally with sequence-terminal rewards.
The defence has to be precise about both halves: what the critic actually did (per-token credit assignment via GAE, variance reduction), what replaces it (empirical baselines from groups of rollouts), why this fits modern LLM RL (sequence-terminal rewards, cheap multi sample per prompt), and where the replacement breaks down (long-horizon credit assignment, small or saturated groups, dense intermediate signals).
Mental model: the critic was a clever piece of machinery solving a problem that mostly disappears when rewards are sequence-terminal. Drop the critic, sample several rollouts per prompt, use their statistics as a baseline. Works great for math and code; less great for long agentic tasks.
What the PPO critic does and why the field tolerated it for so long
GAE per-token credit assignment. The PPO critic V_phi(s) estimates the expected return from any state in the rollout. Generalized Advantage Estimation combines bootstrapped returns from the critic with empirical returns from the rollout to produce per-token advantage estimates:
The lambda parameter trades off bias and variance. Each token gets its own advantage, which the policy gradient uses for fine-grained credit assignment.
Variance reduction as a control variate. The value baseline reduces gradient variance without adding bias (in expectation). For tasks with high-variance returns this matters; the policy gradient with no baseline can be too noisy to learn from.
Why the field tolerated the value head despite its cost. In the early RLHF era (2020-2022), reward signals were per token credible (continuous RM scores applied to whole sequences) and rollouts were expensive (compute was the bottleneck). The cost of doubling memory for the value head was offset by the variance reduction it provided. Production RLHF runs with PPO and a value head were the default through 2023.
What changed. Two structural shifts:
- Reward structure shifted toward sequence-terminal. RLVR (math correctness, code tests), preference-pair training (DPO/SimPO-style), and constitution-following AI labels all produce single scalar rewards per sequence. The per-token credit assignment GAE provided has no per-token signal to attach to.
- Rollout compute became cheaper relative to value-fitting overhead. Modern inference is fast enough that sampling 8 to 16 rollouts per prompt is cheap. The empirical baseline from a group of rollouts has variance reduction properties comparable to GAE without the value-head cost.
The combination meant the critic was paying for a benefit (per-token credit assignment) that the new reward structure could not exploit, while continuing to pay the cost (second model, fitting instability) that scaled with model size.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-R1 (2025) used GRPO for reasoning post-training with verifiable rewards; the recipe is the open reference for GRPO at scale.
- Shao et al., DeepSeek-Math (2024) introduced GRPO; the paper compares against PPO with value head and shows the engineering wins on math reasoning.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you diagnose whether your GRPO run is suffering from saturated groups, and what would you do about it?
Log per-batch the fraction of prompts where the within-group reward variance is below a threshold (typical: 0.01 of mean). Track the trend; a rising fraction over training is the saturation signal. Mitigations: filter low-variance prompts before computing advantages; raise sampling temperature for those prompts; use difficulty-aware prompt selection that targets prompts with predicted intermediate success rates.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing GRPO and RLOO are strictly better than PPO; they trade per-token credit assignment for engineering simplicity and work well only when the reward is sequence-terminal and groups are large enough.
60 second bullets to scan on the way to the call.
The critic's role in PPO: GAE per-token advantages, variance reduction, credit assignment
The critic's costs: second large model, value-fitting instability on text prefixes
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.