What is RLHF, and why is it used after pretraining?

**RLHF** stands for **Reinforcement Learning from Human Feedback**. It's the post training step that takes a raw pretrained LLM, which knows a lot but doesn't know how to be helpful, and aligns it with human preferences. The pipeline has three stages. First, supervised fine-tuning (SFT). You hand the pretrained model a curated dataset of high-quality prompt-response pairs and fine-tune it to mimic that style. This gets the model into the rough shape of an assistant. Second, reward modeling. You collect human preference data: for each prompt, show two model outputs to a human and ask which is better. Train a small model (the reward model) to predict the human's preference. Third, RL fine-tuning. Use reinforcement learning (classically PPO) to update the LLM so its outputs maximize the reward model's score, while a KL penalty keeps it from straying too far from the SFT model. The result is a **model that's not just knowledgeable but actually helpful**.

**RLHF** is the post pretraining alignment process that turned raw language models into useful assistants. It's why ChatGPT in 2022 felt qualitatively different from GPT-3 a year earlier. The canonical pipeline has three stages: 1. Supervised fine-tuning (SFT) on curated demonstrations. Teaches the basic assistant format. 2. Reward model training on preference pairs. Humans see two responses, pick the better one; a model learns to predict that preference as a scalar score. 3. RL fine-tuning (originally PPO, now often DPO or its variants). Update the policy to maximize reward, with a KL leash to the SFT reference. Why it's needed. Pretraining teaches the model to predict the next token of internet text. The next-token objective rewards mimicry, not helpfulness. A pretrained model asked 'what's the capital of France?' might reply with a quiz style question itself, because that pattern appeared in its training data. RLHF closes the gap between 'predict text' and 'do what the user wants'. In 2026, the term has broadened. Original PPO-based RLHF is now one option; DPO is the production default; RLAIF (Anthropic's constitutional methods) replaces some human labels with AI feedback. The shared idea remains: align via preference signal, not just imitation.