What is RLHF, and why is it used after pretraining?

**RLHF** stands for **Reinforcement Learning from Human Feedback**. It's the post training step that aligns a raw pretrained LLM with human preferences. A raw model knows a lot but doesn't know how to be helpful. The pipeline has three stages. First, supervised fine-tuning (SFT). You hand the pretrained model a curated dataset of high-quality prompt-response pairs and fine-tune it to mimic that style. This gets the model into the rough shape of an assistant. Second, reward modeling. You collect human preference data: for each prompt, show two model outputs to a human and ask which is better. Train a second model (the reward model) to predict the human's preference. Third, RL fine-tuning. Use reinforcement learning (classically PPO, Proximal Policy Optimization) to update the LLM so its outputs maximize the reward model's score. A KL penalty (a cost for drifting) keeps it from straying too far from the SFT model. The result is a **model that's not just knowledgeable but helpful**. **Option B is correct.** A is wrong: RLHF runs after pretraining and is not a regularizer. C (a tokenization speed-up) and D (an embedding encoding) have nothing to do with training on human preferences.

**Option B.** **RLHF** is the post pretraining alignment process that turned raw language models into useful assistants. OpenAI says it trained ChatGPT (2022) with RLHF, using the same methods as InstructGPT. The standard pipeline has three stages: 1. Supervised fine-tuning (SFT) on curated demonstrations. It teaches the basic assistant format. 2. Reward model training on preference pairs. Humans see two responses and pick the better one. A model learns to predict that preference as a scalar score. 3. RL fine-tuning, classically with PPO. Update the policy to maximize reward, with a KL leash to the SFT reference. DPO and its variants instead train on the preference pairs with a supervised loss, with no separate reward model or RL loop. Why it's needed. Pretraining teaches the model to predict the next token of internet text. The next-token objective rewards mimicry, not helpfulness. A pretrained model asked 'what's the capital of France?' might reply with a quiz style question itself, because that pattern appeared in its training data. RLHF closes the gap between 'predict text' and 'do what the user wants'. The term has broadened. PPO-based RLHF is one option. DPO is widely used (Llama 3 chose it over PPO). RLAIF, as in Anthropic's Constitutional AI, replaces some human labels with AI feedback. The shared idea remains: align via preference signal, not just imitation. The other options miss this: RLHF is not a pretraining regularizer (A), a tokenization trick (C), or an embedding encoding (D).