A DPO run spikes to NaN around step 400 after a clean start: explain the most likely cause

Short answer·Medium·4.0 · 0·~3 min·Relevant atDatabricks Meta OpenAI

Attempt it

A DPO run trains cleanly for the first few hundred steps, then training loss spikes to NaN around step 400 and never recovers. Diagnose the most likely cause and prescribe three concrete fixes a practitioner would try, in order.

Free · 2 AI evals / day

**DPO loss has a specific failure mode** that does not show up in plain SFT. The loss includes a sigmoid of beta times the difference between the policy's log-probability ratio and the reference's log-probability ratio, computed on chosen versus rejected pairs. When the model encounters a pathological preference pair (one where chosen and rejected are almost identical, or where the label is just wrong), the log-probability difference can grow uncontrollably. Multiply by beta (usually around 0.1) and the exponent inside the sigmoid becomes large. In fp16, which has a narrow exponent range, the math overflows to infinity, then to NaN, and the run is dead. Three fixes, ordered cheapest to most invasive. First, **lower beta** from 0.1 to 0.05 or 0.03. Beta is the amplifier on the dangerous exponent. Halving it usually pulls the math back into the safe range and the run recovers. This is often enough on its own. Second, **add or tighten gradient clipping**. Set max_grad_norm to 1.0 if it was off, or 0.5 if it was already at 1.0. Clipping prevents a single bad batch from moving the weights catastrophically, which buys robustness against any remaining pathological pairs. Third, **switch from fp16 to bf16**. bf16 has the same wide exponent range as fp32 (just less mantissa precision), so it does not overflow at the values that kill fp16. This removes the numerical leg of the failure entirely. Deeper fix: audit the preference dataset. Near-duplicate or contradictory pairs are the source of the spike; dropping them prevents recurrence.

**The cause is DPO logit divergence amplified by beta**, with fp16 overflow as the proximate trigger. The DPO loss includes: $$\mathcal{L}_{\text{DPO}} = -\log \sigma\!\left(\beta \log \frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} - \beta \log \frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}\right)$$ The term inside the sigmoid is the DPO logit. As training progresses, the policy moves away from the reference, and on pathological preference pairs (chosen and rejected near-identical, or label flipped), the policy logratio difference can grow without bound. Beta multiplies it. In fp16, the exponent range is roughly +/-15 in the natural-log sense; a single batch where the DPO logit reaches that magnitude overflows the sigmoid to inf, then NaN, and the run is unrecoverable. Three fixes, ordered cheapest to most invasive. **Lower beta.** Drop from typical 0.1 to 0.05 or 0.03. Beta is the direct amplifier on the dangerous exponent; halving it pulls the math back into the safe range and is often enough on its own. The trade-off is slightly less aggressive preference pressure, which may slow convergence but is preferable to NaN. **Add or tighten gradient clipping.** Set max_grad_norm to 1.0 if it was off, or 0.5 if already at 1.0. Clipping bounds the per-step weight movement from any single batch, so even if the DPO logit spikes again, the gradient norm is capped and the update stays safe. Cheap to add and removes the worst-case failure path. **Switch from fp16 to bf16.** bf16 has the same 8-bit exponent as fp32, covering the same dynamic range, with reduced mantissa precision. The DPO logit can grow to magnitudes that overflow fp16 without overflowing bf16. This eliminates the numerical leg of the failure entirely, at no quality cost on Hopper-class hardware. Deeper fix: audit the preference dataset. Near-duplicate or contradictory pairs are the trigger; embedding-based similarity filtering catches them.

Quick reference

TL;DR

The DPO logit term diverges on a pathological pair, beta amplifies it, fp16 overflows. Fix order: lower beta, add gradient clipping, switch to bf16; then audit the dataset.

Memory aid

Easy to grasp

Picture two students taking a quiz where they compare answers. Usually the teacher gently nudges them when one answer is clearly better. But on a quiz where the two answers are nearly identical (or the label is just wrong about which is better), the teacher's nudge becomes a violent shove because the rule says the more confident you are about a comparison, the harder you push. If the teacher is also stingy with precision (can only express scores between very narrow bounds), one bad shove can break the entire scorekeeping system and the whole class falls apart. The first fix is to make the teacher less aggressive, the second is to cap how hard any single shove can be, and the third is to give the teacher a more flexible scorekeeping format.

Key concepts

DPO Training stability Mixed precision

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: the DPO loss term that diverges, the beta amplifier, why fp16 overflows while bf16 does not, the three ordered fixes, NaN propagation through AdamW state, and the deeper preference-dataset audit.

Side-by-side

Symptom timing	Likely cause	First fix
Flat from step 0	Structural break (LR, wrapping, labels)	Trace optimization path
Clean start, NaN mid-run (this case)	DPO logit divergence + fp16	Lower beta, clip, bf16
Loss drops then plateaus	Capacity or convergence	Check eval; bump rank if eval flat
Train down, eval up	Overfitting	Early stopping; reduce capacity
Oscillating loss	LR too high or small batch	Lower LR, larger batch, more warmup

Real-world examples

Real products, models, and research that use this idea.

TRL DPOTrainer in 2026 defaults to bf16 on Hopper hardware specifically to avoid the fp16 NaN failure mode this question describes.
Llama 4 Maverick preference-tuning recipes ship with beta=0.05 by default after community reports of NaN spikes at beta=0.1 on noisy crowd-sourced pairs.
Axolotl YAML configs for DPO runs set max_grad_norm=1.0 as the default since DPO is more prone to gradient spikes than SFT.
Hugging Face's TRL repo issue history is full of DPO NaN bug reports traced to beta too high, fp16, or contradictory preference pairs.
Mistral Large 3 community DPO fine-tunes in 2026 routinely include an embedding-similarity preprocessing step on the preference dataset to drop near-duplicate chosen/rejected pairs.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does NaN propagate forward indefinitely once it appears in DPO?

QHow do you choose between dropping beta to 0.05 versus 0.03?

QWhat is the embedding-similarity threshold to use when filtering near-duplicate preference pairs?

QDoes switching to bf16 always work, or are there cases where it does not save the run?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Blaming the learning rate when DPO spikes to NaN. The DPO loss has its own amplifier (beta) on a term that can diverge independently of LR, and the right diagnostic is the logit blow-up, not a generic optimization issue.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.