ORPO (Hong et al. 2024) markets itself as 'single stage' fine-tuning. What does it combine into one training pass, what's the actual loss term that makes that possible, and what operational benefit do you get compared to SFT then DPO?
ORPO folds SFT and preference alignment into one pass using an odds-ratio penalty on the SFT loss. No reference model, memory close to plain SFT.
Imagine training a dog. The usual way is two lessons. First you reward it for good tricks until it learns them. Then, separately, you correct it whenever it picks a bad trick over a good one, and to judge that you keep a second untrained dog around as a baseline. ORPO does both lessons at once with a single trainer and no spare dog. While rewarding the good trick, it also nudges the odds away from the bad one in the very same moment. You feed it pairs of good and bad answers, and one training run teaches both the skill and the preference. Fewer lessons, no spare dog to feed, one well-trained result at the end.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Preference alignment used to be a relay race. First you ran supervised fine-tuning on demonstrations so the model produced reasonable answers. Then you ran a second method, usually DPO, on preference pairs so the model learned to favour better answers over worse ones. Two stages, two training jobs, two sets of hyperparameters, and two checkpoints to babysit. The hand-off between them is where teams lose time and introduce bugs.
Most of those preference methods also carry a hidden cost: a frozen reference policy. DPO scores how far the trainable model's preference has moved relative to a frozen copy of the starting model. That copy lives in memory and gets its own forward pass every step. For large models, that is a real footprint and a real slowdown, and it is the single biggest reason a preference run costs more than the SFT run that preceded it.
ORPO, short for Odds Ratio Preference Optimization, asks a sharper question. If supervised fine-tuning already teaches the model good outputs, why not slip the preference signal directly into that same loss and finish in one pass? The insight is that the supervised phase and the alignment phase are not really separate learning problems; they are two pressures on the same output distribution. The answer ORPO found is an odds-ratio penalty that needs no reference policy at all, riding alongside the ordinary cross-entropy.
This deep dive walks the baseline pipeline, the exact loss ORPO optimises, why the odds-ratio form is what kills the reference-model requirement, the memory and operational consequences, and the honest quality trade-off against a well-tuned two-stage chain. By the end you should be able to write the loss from memory and defend when you would and would not reach for it.
The two-stage baseline and its overhead
The classic recipe is supervised fine-tuning followed by a preference stage. SFT trains on (prompt, chosen) pairs with ordinary next-token cross-entropy. The model learns to imitate good demonstrations, which is necessary but not sufficient: imitation teaches what a good answer looks like, not which of two plausible answers is better.
Then comes the preference stage. DPO trains on (prompt, chosen, rejected) triples and pushes the model to assign higher probability to chosen than rejected. To keep the update from drifting too far, DPO measures the policy against a frozen reference policy, which is typically the SFT checkpoint itself. The reference acts as a leash: the loss rewards moving probability toward chosen, but only relative to where the reference already sat.
That reference policy is the quiet tax. It sits in memory beside the trainable model and runs its own forward pass on every batch, so the preference stage can need close to twice the memory and noticeably more compute per step than the SFT stage did. Operationally you also juggle two jobs, two hyperparameter sweeps, and a checkpoint hand-off between stages. Every seam is a place for a bug or a stale artifact to creep in, and the second job cannot start until the first has fully converged.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Concern | ORPO (single-stage) | SFT then DPO (two-stage) |
|---|---|---|
| Training passes | One pass over (prompt, chosen, rejected) | Two passes: SFT then DPO |
| Reference policy | None; penalty is on the trainable model | Frozen reference policy held in memory |
| Memory footprint | Close to plain SFT | Near double during the DPO stage |
| Hyperparameters | One set, plus a small lambda weight | Two sets, tuned per stage |
| Artifacts to manage | One checkpoint to evaluate | SFT checkpoint then DPO checkpoint |
| Quality on heavy preference data | Strong, sometimes a notch below | Often the quality ceiling when tuned |
Real products, models, and research that use this idea.
- Hugging Face TRL ships an ORPOTrainer, used by community fine-tunes of Llama 4 and Mistral bases to skip the separate DPO stage.
- Axolotl exposes ORPO as a one-config training recipe, popular for quick preference tunes of open-weight models on a single 80GB GPU.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the odds ratio, rather than the probability ratio, let ORPO drop the reference policy?
Trace what DPO's reference policy is for: it anchors the policy ratio so updates stay near the base. ORPO's odds ratio is a self-contained contrast between chosen and rejected on one model, so no anchor is needed.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Describing ORPO as just DPO without a reference model. It also fuses the SFT stage, and its penalty uses an odds ratio, not the policy ratio DPO uses.
60 second bullets to scan on the way to the call.
Why preference pipelines were traditionally two stages
What ORPO fuses into a single training pass
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.