Modern alignment usually runs in two stages. First **SFT** teaches the model to follow instructions on good examples. Then a preference step, like DPO, nudges it toward answers humans prefer.
DPO needs a frozen copy of the model, called the reference policy. During training it runs both the trainable model and this frozen copy, then compares their outputs. That second copy eats memory and forces you to finish SFT before you even start preference tuning.
**ORPO** collapses both stages into one. Its loss adds two things together: the normal SFT cross-entropy on the chosen answer, plus an odds-ratio penalty that pushes the chosen answer's odds above the rejected answer's odds.
Because that penalty only looks at the model being trained, ORPO needs no reference copy. You run a single training pass on (chosen, rejected) pairs and get instruction-following plus preference alignment together.
The payoff is simplicity and memory. One stage, one model in memory, roughly the cost of plain SFT, with alignment baked in.
**ORPO** folds supervised fine-tuning and preference alignment into a single training stage. Its loss is the standard SFT cross-entropy on the chosen response, plus a log-sigmoid odds-ratio penalty.
That penalty compares the chosen response against the rejected one inside the same batch. It raises the odds the policy assigns to the preferred answer relative to the dispreferred answer. Crucially, it reads only from the policy being trained.
DPO, by contrast, needs a frozen reference policy. It scores log-probability ratios between the trainable model and that frozen copy. So DPO carries a second model through training and typically follows a separate SFT warm-up stage.
ORPO removes the reference policy entirely. No second forward pass, no frozen weights, no warm-up dependency. The memory budget sits close to plain SFT, which is the headline practical win.
The trade-off is control. DPO's reference anchor limits how far the policy drifts from its starting point. ORPO leans on its lambda weight and the SFT term to stay grounded instead. For many open-weight bases that is enough, which is why ORPO became a popular one-shot alignment recipe.
Quick reference
TL;DR
ORPO adds an odds-ratio penalty straight onto the SFT loss, so it aligns and instruction-tunes in one pass with no reference model.
Picture training a chef. The normal recipe (SFT) teaches the chef to cook the dish you like. Preference tuning then nudges them toward the better of two plates. Older methods like DPO keep a frozen copy of the old chef in the kitchen to compare against, which costs a second oven. ORPO skips that. It writes one combined recipe: cook the good dish, and at the same time push the odds of the good dish above the bad one. No frozen copy, no second oven, one cooking session. You get a chef who both knows the dish and prefers the better plate, trained in a single sitting at roughly the memory of plain cooking lessons.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You already have a high quality SFT checkpoint and only want a light preference nudge from it DPO from that checkpoint gives explicit reference anchoring; restarting combined training throws away the warm-up you paid for.
Alignment that must provably stay close to the base distribution for safety review DPO's reference term bounds divergence explicitly; ORPO relies on a tuned lambda, which is harder to certify.
Only thumbs up or thumbs down labels exist, with no paired chosen versus rejected data ORPO's odds-ratio term needs pairwise preferences; KTO handles unpaired binary feedback instead.
The actual problem is missing knowledge rather than wrong behavior or style No preference method changes facts; retrieval or RAG addresses freshness, not output preference.
Quick scan · 2–4 min
Everything important, quickly.
4 min: DPO two-stage and reference policy + the ORPO loss decomposition + why removing the reference enables single-stage + drift control via lambda + why each distractor describes a different method.
Side-by-side
Aspect
DPO
ORPO
Stages
SFT warm-up, then preference
Single combined stage
Reference policy
Required, frozen copy
None
Loss shape
Log-ratio vs reference
SFT cross-entropy plus odds-ratio penalty
Memory at train time
Two models in memory
Roughly plain SFT
Drift control
Explicit reference anchor
Lambda weight plus retained SFT term
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face TRL ships an ORPOTrainer used to align open-weight bases like Llama 4 and Mistral variants in a single run.
2Axolotl and LLaMA-Factory both expose ORPO as a config flag, popular for low-budget single-node alignment of 7B to 13B models.
3Practitioners aligning DeepSeek V4 distillations report ORPO matching DPO quality while cutting the pipeline from two jobs to one.
4Unsloth tutorials demonstrate ORPO fine-tuning of small open models on a single consumer GPU, leaning on its reference-free memory profile.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWithout a reference policy, what stops ORPO from drifting arbitrarily far from the base model?
Sign in or go Pro to see the approach for this follow-up.
QHow does the odds-ratio formulation differ mathematically from DPO's log-probability ratio?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you still prefer the two-stage DPO pipeline over single-stage ORPO?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming ORPO drops the SFT loss or hides a reference model. It keeps cross-entropy and adds an odds-ratio term, with no reference policy at all.