GRPO (Group Relative Policy Optimization)
Also known as: Group Relative Policy Optimization
PPO without the critic: advantages are computed by ranking multiple sampled responses against each other.
A simplified RL algorithm popularized by DeepSeek-R1 that eliminates the value-function critic by computing advantages relative to a group of sampled responses. Cheaper and more stable than PPO for reasoning-model training.
In practice
Powers DeepSeek-R1 and increasingly the reasoning-model training stack. Cutting-edge interview topic for alignment-focused roles.
How it compares
PPO uses a learned value function as critic; GRPO uses the group mean reward, dropping the critic entirely.
Related topics
Related terms
RLHF (Reinforcement Learning from Human Feedback)
Train a reward model from human preference pairs, then RL-fine-tune the LLM against that reward.
DPO (Direct Preference Optimization)
Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.
Reasoning Model
An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.
Pre-training
Self-supervised next-token training on a huge unlabeled corpus, producing the base model.
Supervised Fine-Tuning (SFT)
Fine-tune the base model on (prompt, ideal response) pairs; the first post-training step before alignment.
PPO (Proximal Policy Optimization)
An RL algorithm with clipped updates: the classic engine of RLHF, now often replaced by DPO.