Preference Learning

Sequence the forward passes and loss steps inside a single DPO training step

Reward model in RLHF: what does it score, and on what data does it train?

Preference pair: name the two pieces and what the label expresses.

SimPO: what does it drop vs DPO, and what's the cost win?

ORPO: what does it combine into one stage, and why is that attractive?

ORPO vs DPO: which architectural difference enables 'single stage' training?

When would you pick KTO over DPO?

DPO: what's the closed form trick that lets it skip the reward model?

DPO vs RLHF: pick the most accurate operational difference

DPO failure modes: large β, small β, noisy preferences