RLHF (Reinforcement Learning from Human Feedback)
Also known as: Reinforcement Learning from Human Feedback
Train a reward model from human preference pairs, then RL-fine-tune the LLM against that reward.
A training procedure that aligns LLMs with human preferences. A reward model is trained on human comparison data, then the LLM is fine-tuned via PPO or DPO to maximize the learned reward signal.
In practice
Underpins ChatGPT, Claude, and Gemini's helpfulness. Senior interviews test reward-hacking, KL penalties, and why DPO is gradually replacing PPO.
How it compares
Supervised fine-tuning teaches the model what to say; RLHF teaches it which of several responses is preferred.
Comparisons that include RLHF (Reinforcement Learning from Human Feedback)
Related topics
Related terms
DPO (Direct Preference Optimization)
Skip the reward model and PPO: fine-tune directly on preferred-vs-rejected response pairs.
Reasoning Model
An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.
Pre-training
Self-supervised next-token training on a huge unlabeled corpus, producing the base model.
Supervised Fine-Tuning (SFT)
Fine-tune the base model on (prompt, ideal response) pairs; the first post-training step before alignment.
PPO (Proximal Policy Optimization)
An RL algorithm with clipped updates: the classic engine of RLHF, now often replaced by DPO.
GRPO (Group Relative Policy Optimization)
PPO without the critic: advantages are computed by ranking multiple sampled responses against each other.