ORPO: what does it combine into one stage, and why is that attractive?

Attempt it

ORPO (Hong et al. 2024) markets itself as 'single-stage' fine-tuning. What does it combine into one training pass, what's the actual loss term that makes that possible, and what operational benefit do you get compared to SFT-then-DPO?

Free · 2 AI evals / day

Quick reference

TL;DR

ORPO folds SFT and preference alignment into one pass using an odds-ratio penalty on the SFT loss. No reference model, memory close to plain SFT.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine training a dog. The usual way is two lessons. First you reward it for good tricks until it learns them. Then, separately, you correct it whenever it picks a bad trick over a good one, and to judge that you keep a second untrained dog around as a baseline. ORPO does both lessons at once with a single trainer and no spare dog. While rewarding the good trick, it also nudges the odds away from the bad one in the very same moment. You feed it pairs of good and bad answers, and one training run teaches both the skill and the preference. Fewer lessons, no spare dog to feed, one well-trained result at the end.

Key concepts

Odds Ratio Preference Optimization Direct Preference Optimization Preference learning Reference policy

Quick scan · 2–4 min

Everything important, quickly.

4 min: two-stage baseline + what ORPO fuses + the two-term loss + why the odds ratio removes the reference policy + memory and operational wins + quality trade-off versus staged DPO.

Side-by-side

Concern	ORPO (single-stage)	SFT then DPO (two-stage)
Training passes	One pass over (prompt, chosen, rejected)	Two passes: SFT then DPO
Reference policy	None; penalty is on the trainable model	Frozen reference policy held in memory
Memory footprint	Close to plain SFT	Near double during the DPO stage
Hyperparameters	One set, plus a small lambda weight	Two sets, tuned per stage
Artifacts to manage	One checkpoint to evaluate	SFT checkpoint then DPO checkpoint
Quality on heavy preference data	Strong, sometimes a notch below	Often the quality ceiling when tuned

Real-world examples

Real products, models, and research that use this idea.

Hugging Face TRL ships an ORPOTrainer, used by community fine-tunes of Llama 4 and Mistral bases to skip the separate DPO stage.
Axolotl exposes ORPO as a one-config training recipe, popular for quick preference tunes of open-weight models on a single 80GB GPU.
Teams iterating on small preference sets adopt ORPO over SFT then DPO to cut wall-clock from two jobs to one before benchmarking against Claude Opus 4.7.
Unsloth documents ORPO runs that fit in roughly the memory of plain SFT, since no frozen reference policy is held resident.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the odds ratio, rather than the probability ratio, let ORPO drop the reference policy?

Sign in or go Pro to see the approach for this follow-up.

QHow would you choose and tune the lambda weight on the odds-ratio term?

Sign in or go Pro to see the approach for this follow-up.

QWhen would you migrate an ORPO model to a staged SFT then DPO pipeline?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.