ORPO: what does it combine into one stage, and why is that attractive?

The standard way to align a model on preferences is two separate steps. First, supervised fine-tuning (**SFT**) teaches the model good answers from demonstrations. Then a preference method like DPO teaches it to prefer chosen answers over rejected ones. Two steps means two training jobs, two sets of knobs to tune, and two checkpoints to track. It also means most preference methods keep a frozen copy of the original model around as a baseline, which doubles the memory for that part. ORPO collapses both steps into one. You train once on data shaped as (prompt, chosen, rejected). The loss has two pieces added together: the normal SFT loss on the chosen answer, plus an odds-ratio penalty that pushes the chosen answer's odds above the rejected one. Because the penalty compares chosen against rejected inside the same example, ORPO needs no frozen baseline model. Memory stays close to plain SFT. The payoff is operational simplicity. One job, one set of hyperparameters, one checkpoint. The catch is that on heavy preference tasks, a carefully tuned two-stage pipeline can still edge it out.

**ORPO** (Odds Ratio Preference Optimization) merges supervised fine-tuning and preference alignment into one training pass. You drop the separate DPO stage entirely. The data is (prompt, chosen, rejected) triples. The loss adds two terms together. $$\mathcal{L}_\text{ORPO} = \mathcal{L}_\text{SFT} + \lambda \cdot \mathcal{L}_\text{OR}$$ The first term is plain cross-entropy on the chosen response. The second is the odds-ratio term. It applies a log-sigmoid to the log ratio of the odds of chosen over rejected. That nudges the model toward chosen and away from rejected within the same step. Why it stays single-stage: the penalty is computed between the two responses on the trainable policy itself. There is no frozen reference policy and no extra forward pass per step. So memory and compute land close to plain SFT. The operational win is concrete. One training job replaces two. One hyperparameter set replaces two. One checkpoint goes to eval instead of a chain. The trade is that on heavy preference tuning, a well-tuned two-stage pipeline can still score higher. For most teams, the simplicity is worth it.