GRPO (Group Relative Policy Optimization)
Also known as: Group Relative Policy Optimization
PPO without the critic: advantages are computed by ranking multiple sampled responses against each other.
A simplified RL algorithm popularized by DeepSeek-R1 that eliminates the value-function critic by computing advantages relative to a group of sampled responses. Cheaper and more stable than PPO for reasoning-model training.
In practice
Powers DeepSeek-R1 and increasingly the reasoning-model training stack. Cutting-edge interview topic for alignment-focused roles.
How it compares
PPO uses a learned value function as critic; GRPO uses the group mean reward, dropping the critic entirely.
Related topics
Questions that mention this term
Related terms
DPO (Direct Preference Optimization)
Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.
PPO (Proximal Policy Optimization)
An RL algorithm with clipped updates: the classic engine of RLHF, now often replaced by DPO.
Pre-training
Self-supervised next-token training on a huge unlabeled corpus, producing the base model.
Reasoning Model
An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.
RLHF (Reinforcement Learning from Human Feedback)
Train a reward model from human preference pairs, then RL-fine-tune the LLM against that reward.
Supervised Fine-Tuning (SFT)
Fine-tune the base model on (prompt, ideal response) pairs; the first post-training step before alignment.