ORPO vs DPO: which architectural difference enables 'single-stage' training?

Modern alignment usually runs in two stages. First **SFT** teaches the model to follow instructions on good examples. Then a preference step, like DPO, nudges it toward answers humans prefer. DPO needs a frozen copy of the model, called the reference policy. During training it runs both the trainable model and this frozen copy, then compares their outputs. That second copy eats memory and forces you to finish SFT before you even start preference tuning. **ORPO** collapses both stages into one. Its loss adds two things together: the normal SFT cross-entropy on the chosen answer, plus an odds-ratio penalty that pushes the chosen answer's odds above the rejected answer's odds. Because that penalty only looks at the model being trained, ORPO needs no reference copy. You run a single training pass on (chosen, rejected) pairs and get instruction-following plus preference alignment together. The payoff is simplicity and memory. One stage, one model in memory, roughly the cost of plain SFT, with alignment baked in.

**ORPO** folds supervised fine-tuning and preference alignment into a single training stage. Its loss is the standard SFT cross-entropy on the chosen response, plus a log-sigmoid odds-ratio penalty. That penalty compares the chosen response against the rejected one inside the same batch. It raises the odds the policy assigns to the preferred answer relative to the dispreferred answer. Crucially, it reads only from the policy being trained. DPO, by contrast, needs a frozen reference policy. It scores log-probability ratios between the trainable model and that frozen copy. So DPO carries a second model through training and typically follows a separate SFT warm-up stage. ORPO removes the reference policy entirely. No second forward pass, no frozen weights, no warm-up dependency. The memory budget sits close to plain SFT, which is the headline practical win. The trade-off is control. DPO's reference anchor limits how far the policy drifts from its starting point. ORPO leans on its lambda weight and the SFT term to stay grounded instead. For many open-weight bases that is enough, which is why ORPO became a popular one-shot alignment recipe.