Preference Learning

Sequence the forward passes and loss steps inside a single DPO training step

Reward model in RLHF: what does it score, and on what data does it train?

What are the two pieces of a preference pair, and what does the label express?

What does SimPO drop compared with DPO, and what cost does that save?

What does ORPO combine into one stage, and why?

Which architectural difference between ORPO and DPO enables single stage training?

When would you use KTO over DPO?

DPO: what's the closed form trick that lets it skip the reward model?

What is the operational difference between DPO and RLHF?

What goes wrong in DPO with large β, small β, and noisy preferences?