Dpo

Sequence the forward passes and loss steps inside a single DPO training step

Order the steps in building a 5k-pair DPO dataset on helpfulness.

Preference pair: name the two pieces and what the label expresses.

Loss spikes the moment warmup ends: which hyperparameter is the prime suspect?

A DPO run spikes to NaN around step 400 after a clean start: explain the most likely cause

Match each preference-tuning variant to its design choice on noisy labels

SimPO: what does it drop vs DPO, and what's the cost win?

Spot the error in this description of SFT loss masking

Where do SFT and DPO sit in the classical RLHF pipeline?

ORPO: what does it combine into one stage, and why is that attractive?

ORPO vs DPO: which architectural difference enables 'single-stage' training?

Match training regime to sensible peak LR range

Fill in typical peak LRs for full FT, LoRA, and DPO

When would you pick KTO over DPO?

DPO: what's the closed-form trick that lets it skip the reward model?

DPO vs RLHF: pick the most accurate operational difference

DPO failure modes: large β, small β, noisy preferences

PPO vs DPO, what's the practical difference?