PPO (Proximal Policy Optimization)
Also known as: Proximal Policy Optimization
An RL algorithm with clipped updates: the classic engine of RLHF, now often replaced by DPO.
A reinforcement learning algorithm that constrains policy updates with a clipped objective to prevent destructive large steps. The original RL algorithm used in InstructGPT/ChatGPT's RLHF; increasingly replaced by DPO for alignment.
In practice
Foundational alignment knowledge. Senior interviews probe its trust region intuition and why teams move to DPO/GRPO.
How it compares
PPO trains a separate reward model and runs an RL loop; DPO collapses both into one supervised loss.
Comparisons that include PPO (Proximal Policy Optimization)
Related topics
Related terms
DPO (Direct Preference Optimization)
Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.
GRPO (Group Relative Policy Optimization)
PPO without the critic: advantages are computed by ranking multiple sampled responses against each other.
Pre-training
Self-supervised next-token training on a huge unlabeled corpus, producing the base model.
Reasoning Model
An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.
RLHF (Reinforcement Learning from Human Feedback)
Train a reward model from human preference pairs, then RL-fine-tune the LLM against that reward.
Supervised Fine-Tuning (SFT)
Fine-tune the base model on (prompt, ideal response) pairs; the first post-training step before alignment.