State what RLHF adds that pretraining alone cannot provide

Pretraining teaches a model to predict the next token (a word piece) in huge amounts of internet text. The model becomes fluent and learns many facts. But its only goal is to sound like the internet, and the internet contains wrong answers, rude text, and pages that never answer anything. Ask a pretrained model a question and it may write more questions. **RLHF** (reinforcement learning from human feedback) adds what pretraining cannot: a signal about which answers people want. People compare two of the model's own answers and pick the better one. A reward model (a model that scores answers) learns from these choices. Then the language model is trained to give answers that score high, while it stays close to its starting point. This changes the goal from "likely text" to "preferred answer". In OpenAI's InstructGPT work, people preferred answers from a 1.3B model trained with demonstrations and RLHF over answers from the 175B GPT-3 model. RLHF does not add much new knowledge. It shapes how the model uses what it already knows.

Pretraining maximizes the likelihood of internet text, so it learns to imitate whatever people wrote, good or bad. The InstructGPT paper calls this objective misaligned: predicting the next token on a webpage is not the same as following the user's instructions helpfully and safely. **RLHF** adds three things pretraining cannot give. The first is a target defined by human judgment. Labelers compare responses, and a reward model learns to score whole answers by how much people prefer them. The second is feedback on the model's own outputs. Pretraining only sees text written by others. RLHF samples from the model and pushes down its own specific failures, such as made-up facts or ignored instructions. The third is a way past the quality of demonstrations. People can judge two answers better than they can write a perfect one, so the model is not capped by the best human writer. $$\max_\theta \; \mathbb{E}_{y \sim \pi_\theta}[r_\phi(x,y)] - \beta\, D_{KL}(\pi_\theta \Vert \pi_{\text{ref}})$$ The KL term keeps the model close to its starting point, so it keeps its pretrained knowledge. The effect is large and cheap. Labelers preferred the 1.3B InstructGPT over the 175B GPT-3, and the RL run cost under 2% of GPT-3's pretraining compute. RLHF shapes behavior; it adds little knowledge.