Glossary · Training

PPO (Proximal Policy Optimization)

Also known as: Proximal Policy Optimization

An RL algorithm with clipped updates: the classic engine of RLHF, now often replaced by DPO.

A reinforcement learning algorithm that constrains policy updates with a clipped objective to prevent destructive large steps. The original RL algorithm used in InstructGPT/ChatGPT's RLHF; increasingly replaced by DPO for alignment.

In practice

Foundational alignment knowledge. Senior interviews probe its trust region intuition and why teams move to DPO/GRPO.

How it compares

vs. DPO (Direct Preference Optimization) →Deep comparison →

PPO trains a separate reward model and runs an RL loop; DPO collapses both into one supervised loss.

Comparisons that include PPO (Proximal Policy Optimization)

DPO vs PPODPO turns RLHF into supervised learning on preference pairs, no reward model needed. PPO still wins on some tasks but is trickier to tune.

Related terms

RLHF (Reinforcement Learning from Human Feedback)

Train a reward model from human preference pairs, then RL-fine-tune the LLM against that reward.

DPO (Direct Preference Optimization)

Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.

Reasoning Model

An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.

Pre-training

Self-supervised next-token training on a huge unlabeled corpus, producing the base model.

Supervised Fine-Tuning (SFT)

Fine-tune the base model on (prompt, ideal response) pairs; the first post-training step before alignment.

GRPO (Group Relative Policy Optimization)

PPO without the critic: advantages are computed by ranking multiple sampled responses against each other.