What does the beta parameter control in DPO, and what are the failure modes at extremes?

In DPO, beta sets how strongly preference differences influence training relative to reference anchoring. It affects how aggressively the policy moves toward preferred answers. If beta is too low, updates are weak and the model barely changes. You get little preference improvement. If beta is too high, updates can become overly constrained around the reference behavior, making learning brittle or slow. In practice, both extremes hurt final quality. So beta is a balance parameter. You tune it to get meaningful preference gains without collapsing learning dynamics or over-freezing the policy near its starting point.

In DPO, **beta** scales the log-ratio term that links policy preference to the reference model. Conceptually, it sets how sharp the optimization is when distinguishing preferred vs dispreferred responses under the anchor. At very low beta, the objective becomes too soft; gradients are small and you underfit preference data. At very high beta, the policy can stay overly tied to reference behavior and fail to move enough toward preferred outputs, or optimization can become unstable depending on data quality and implementation details. Interview-ready framing: beta is the main sensitivity knob for the DPO trade-off between adaptation and anchoring. You tune it with validation win-rate, refusal calibration, and capability slices, not by fixed folklore values.