Drag each answer to line up with its matching prompt
Trains a separate value/critic network
PPO
Normalizes rewards across a group of completions for the same prompt
PPO
Better when per token credit assignment matters without dense verifiable signals
GRPO
Lower memory footprint by skipping the critic
GRPO
Natural fit when end of sequence verifiable rewards are reliable
GRPO
PPO keeps a critic for dense credit assignment on fuzzy rewards; GRPO drops the critic, normalizes rewards across sampled completion groups, and fits sparse verifiable outcome rewards on math and code.
Imagine grading a class test two ways. PPO is like having a teaching assistant watch every student's scratch work and estimate how good each partial step is — that takes extra effort but helps on open-ended essays. GRPO is like giving five students the same hard problem, then grading each only relative to the others in that group who got more or less right at the end. You skip the scratch-work watcher, save memory, and it works great when the final answer is objectively checkable.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Post-training for reasoning models revived a practical question: do you still need PPO's critic when rewards come from unit tests and answer checkers? GRPO's answer — compare completions from the same prompt and drop the value network — is now the default in open reasoning recipes. But PPO is not obsolete.
This card maps five structural traits to the right algorithm and explains when each training stack is the rational choice.
PPO: critic-based credit assignment
PPO trains two networks: the policy (the LLM) and a critic that predicts expected cumulative reward from the current state. During RL, advantages guide how much to reinforce each action. When rewards are noisy, delayed, or shaped across tokens, a good critic reduces variance and stabilizes learning.
Classic RLHF for chat alignment uses PPO with a reward model scored on full responses. The reward is not a simple unit test — it is a learned preference signal. Per-token credit assignment is approximate (reward is often applied at EOS or via KL shaping), but the critic still provides a learned baseline that tracks the evolving policy.
Cost: at LLM scale, the critic is often another large transformer head or a separate model — memory, optimizer states, and synchronization overhead. That cost is justified when no cheap verifier exists.
Match: "Trains a separate value/critic network" → PPO. "Better when per-token credit assignment matters without dense verifiable signals" → PPO — fuzzy alignment rewards benefit from a learned baseline even when EOS reward is sparse.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Trait | PPO | GRPO |
|---|---|---|
| Critic / value network | Yes — separate head | No — group mean baseline |
| Reward type fit | Learned preferences, dense shaping | Verifiable EOS outcomes |
| Memory at LLM scale | Higher (policy + critic) | Lower (policy only) |
| Samples per prompt | Often 1+ with replay | Group required (multiple completions) |
| Reasoning RLVR default | Less common | Dominant in 2024-2026 open recipes |
Real products, models, and research that use this idea.
- DeepSeek-R1 post-training uses GRPO with verifiable math and code rewards at scale.
- Classic ChatGPT RLHF stages used PPO with a learned reward model for preference alignment.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does GRPO relate to REINFORCE with a leave-one-out baseline?
Group mean as control variate — compare variance reduction vs critic-based baselines on sparse rewards.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming GRPO replaces PPO everywhere — PPO still wins when rewards are learned preferences without cheap verifiers at every step.
60 second bullets to scan on the way to the call.
State what the PPO critic estimates
Explain group-relative advantage in GRPO in one sentence
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.