Same topic, related formats. Practice these next.
DPO is the brittle baseline; cDPO adds a noise prior, IPO swaps log-sigmoid for a squared margin, KTO drops paired data, and ORPO folds preference into SFT.
Picture a teacher grading essays. The first method, DPO, treats every comparison as gospel: if student A's essay was marked better than B's, push hard on that judgement. That works when graders are reliable but goes wrong when some marks are flipped by mistake. The cousin methods each fix a different worry. One assumes graders sometimes flip the answer and softens the pressure. Another grades by how far apart the essays are rather than which one wins, so close calls do not get punished. A third throws out paired comparisons entirely and just reads thumbs-up or thumbs-down notes one essay at a time. The last one folds the grading right into the main writing class so there is no separate marking stage. Same goal, different beliefs about the graders.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: DPO's log-sigmoid as the baseline, cDPO's noise mixture, IPO's squared margin, KTO's binary feedback, ORPO's reference-free SFT fold-in, and how to pick by data shape and pipeline budget.
| Variant | What it changes | When to pick |
|---|---|---|
| DPO | Baseline log-sigmoid on pairs | Clean paired data, baseline alignment |
| cDPO | Adds label-noise prior to loss | Noisy or low-confidence pairwise labels |
| IPO | Squared loss against target margin | Overfit or unstable DPO runs on small data |
| KTO | Per-sample binary instead of pairs | Thumbs-up / thumbs-down telemetry |
| ORPO | Folds preference into SFT loss | Want one-stage pipeline, no reference model |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating cDPO, IPO, and KTO as drop-in replacements when each one targets a different failure mode of DPO. The choice should follow the data you have, not whichever variant trended last.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.