Match each alignment method to its defining property
Drag each answer to line up with its matching prompt
DPO
Group relative policy updates using sampled candidate sets
KTO
Direct preference optimization from chosen vs rejected pairs
ORPO
Optimization from unpaired binary feedback signals
GRPO
Odds ratio based preference objective without separate reference model
DPO learns from chosen vs rejected pairs, KTO from single answers labeled good or bad, ORPO adds an odds-ratio term to SFT with no reference model, and GRPO scores sampled groups online with no critic.
Think of four ways to coach a cook. The first coach tastes two dishes made for the same order and says which one is better. The second coach tastes one dish at a time and gives only a thumbs up or a thumbs down. Both keep the cook's old recipe book open, so the cook does not drift too far from it. The third coach teaches the recipe and, in the same lesson, points at a bad version and says 'less like that', with no old book needed. The fourth coach asks the cook to make four fresh versions right now, scores each one, and tells the cook to copy whatever beat the group average. These are DPO, KTO, ORPO and GRPO. The real difference is what each coach needs: pairs, single verdicts, pairs during the lesson, or a scorer.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
DPO, KTO, ORPO and GRPO all push a language model toward preferred behavior after pre-training, but each one consumes a different kind of training signal. DPO learns from pairs of chosen and rejected answers. KTO learns from single answers labeled desirable or undesirable. ORPO adds an odds-ratio preference term to the SFT loss and needs no reference model. GRPO is an online RL method that samples a group of answers per prompt and scores each one against the group.
The defining property of each method follows from two questions. What does one training example look like? Which extra models must sit in memory during training? The answers to both also tell you which method fits a given dataset and compute budget.
What one training example looks like
The four methods differ first in what one training record contains.
- DPO: a prompt x, a chosen answer y_w and a rejected answer y_l. Both answers must belong to the same prompt.
- KTO: a prompt x, one answer y, and a binary label: desirable or undesirable. No second answer is needed.
- ORPO: the same triple as DPO. The chosen answer also serves as the SFT target.
- GRPO: only a prompt. The model generates G answers during training, and a reward function scores them.
The first three methods are offline. They read a fixed dataset that was collected before training started. GRPO is online. Its training data comes from the current policy, so the data changes as the model changes. This difference drives most of the cost and stability trade-offs between the methods.
The records also show which conversions are possible. A DPO pair splits into two KTO examples: the chosen answer becomes desirable and the rejected answer becomes undesirable. The KTO paper builds its training data this way from preference datasets. The reverse is harder. Thumbs-up and thumbs-down logs become DPO pairs only when the same prompt has both a good and a bad answer, and production logs rarely have that.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 used DPO with beta 0.1 on preference pairs across six post-training rounds, after finding that DPO needed less compute than PPO at large scale.
- Mistral-ORPO-β (7B) was trained with ORPO on UltraFeedback in a single stage, with lambda 0.1 and no reference model.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you pick between DPO and KTO in a new product stack?
Compare the feedback format, labeling cost and class balance. Pairs on the same prompt suit DPO; single-answer labels, even heavily unbalanced ones, suit KTO through its separate desirable and undesirable weights.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling GRPO a pairwise preference method, or assuming ORPO keeps a reference model like DPO, mixes up what each method consumes and which models must sit in memory.
60 second bullets to scan on the way to the call.
One training record for DPO, KTO, ORPO and GRPO
Role of beta in the DPO loss
Primary sources. Browse if you want the original framing.
- Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization
- Hong et al., ORPO: Monolithic Preference Optimization without Reference Model
- Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Same topic, related formats. Practice these next.