A DPO run spikes to NaN around step 400 after a clean start: explain the most likely cause

Short answer·Medium·4.0 · 0·~3 min·Asked atOpenAI Rephrase Ai Swiggy·Relevant atCoreweave Databricks Fireworks Ai Lambda Labs

Attempt it

A DPO run trains cleanly for the first few hundred steps, then training loss spikes to NaN around step 400 and never recovers. Diagnose the most likely cause and prescribe three concrete fixes a practitioner would try, in order.

Free · 2 AI evals / day

**DPO loss has a specific failure mode** that does not show up in plain SFT. The loss includes a sigmoid of beta times the difference between the policy's log-probability ratio and the reference's log-probability ratio, computed on chosen versus rejected pairs. When the model encounters a pathological preference pair (one where chosen and rejected are almost identical, or where the label is just wrong), the log-probability difference can grow uncontrollably. Multiply by beta (usually around 0.1) and the exponent inside the sigmoid becomes large. In fp16, which has a narrow exponent range, the math overflows to infinity, then to NaN, and the run is dead. Three fixes, ordered cheapest to most invasive. First, **lower beta** from 0.1 to 0.05 or 0.03. Beta is the amplifier on the dangerous exponent. Halving it usually pulls the math back into the safe range and the run recovers. This is often enough on its own. Second, **add or tighten gradient clipping**. Set max_grad_norm to 1.0 if it was off, or 0.5 if it was already at 1.0. Clipping prevents a single bad batch from moving the weights catastrophically, which buys robustness against any remaining pathological pairs. Third, **switch from fp16 to bf16**. bf16 has the same wide exponent range as fp32 (just less mantissa precision), so it does not overflow at the values that kill fp16. This removes the numerical leg of the failure entirely. Deeper fix: audit the preference dataset. Near-duplicate or contradictory pairs are the source of the spike; dropping them prevents recurrence.

**The cause is DPO logit divergence amplified by beta**, with fp16 overflow as the proximate trigger. The DPO loss includes: $$\mathcal{L}_{\text{DPO}} = -\log \sigma\!\left(\beta \log \frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} - \beta \log \frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}\right)$$ The term inside the sigmoid is the DPO logit. As training progresses, the policy moves away from the reference, and on pathological preference pairs (chosen and rejected near-identical, or label flipped), the policy logratio difference can grow without bound. Beta multiplies it. In fp16, the exponent range is roughly +/-15 in the natural-log sense; a single batch where the DPO logit reaches that magnitude overflows the sigmoid to inf, then NaN, and the run is unrecoverable. Three fixes, ordered cheapest to most invasive. **Lower beta.** Drop from typical 0.1 to 0.05 or 0.03. Beta is the direct amplifier on the dangerous exponent; halving it pulls the math back into the safe range and is often enough on its own. The trade-off is slightly less aggressive preference pressure, which may slow convergence but is preferable to NaN. **Add or tighten gradient clipping.** Set max_grad_norm to 1.0 if it was off, or 0.5 if already at 1.0. Clipping bounds the per-step weight movement from any single batch, so even if the DPO logit spikes again, the gradient norm is capped and the update stays safe. Cheap to add and removes the worst-case failure path. **Switch from fp16 to bf16.** bf16 has the same 8-bit exponent as fp32, covering the same dynamic range, with reduced mantissa precision. The DPO logit can grow to magnitudes that overflow fp16 without overflowing bf16. This eliminates the numerical leg of the failure entirely, at no quality cost on Hopper-class hardware. Deeper fix: audit the preference dataset. Near-duplicate or contradictory pairs are the trigger; embedding-based similarity filtering catches them.

TL;DR

The DPO logit term diverges on a pathological pair, beta amplifies it, fp16 overflows. Fix order: lower beta, add gradient clipping, switch to bf16; then audit the dataset.

Memory aid

Easy to grasp

Picture two students taking a quiz where they compare answers. Usually the teacher gently nudges them when one answer is clearly better. But on a quiz where the two answers are nearly identical (or the label is just wrong about which is better), the teacher's nudge becomes a violent shove because the rule says the more confident you are about a comparison, the harder you push. If the teacher is also stingy with precision (can only express scores between very narrow bounds), one bad shove can break the entire scorekeeping system and the whole class falls apart. The first fix is to make the teacher less aggressive, the second is to cap how hard any single shove can be, and the third is to give the teacher a more flexible scorekeeping format.

Key concepts

DPO Training stability Mixed precision

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Mid-run NaN spikes in DPO are one of the most common production failures in preference tuning, and they are also one of the most diagnosable once you understand the loss structure. The clean start tells you the wiring is correct; the spike tells you something specific overwhelmed something specific. For DPO the suspects are always the same: the logit term inside the sigmoid, beta as its amplifier, and fp16's narrow exponent range as the trigger.

The scenario in this question (clean for hundreds of steps, then sudden NaN at step 400) is the textbook signature. The pattern excludes structural breaks (those produce flat loss from step 0), excludes overfitting (that produces a train-eval divergence, not a NaN), and excludes generic LR issues (those usually produce oscillation, not a single catastrophic spike). The remaining suspects are the DPO logit, beta, and the numeric format.

The rest of this section walks the loss mechanism in detail, explains how beta amplifies the divergence, why fp16 overflows where bf16 does not, the three layered fixes in cheapest to most invasive order, and the dataset audit that prevents recurrence beyond any numerical band-aid.

The DPO loss term that diverges

The DPO loss is the negative log-sigmoid of beta times the difference of two log-probability ratios:

\mathcal{L}_{\text{DPO}} = -\log \sigma\!\left(\beta \log \frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} - \beta \log \frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}\right)

The argument inside the sigmoid is the DPO logit. Define it as z = beta * (log_ratio_chosen - log_ratio_rejected), where each log-ratio is the policy log-prob minus the reference log-prob. On a clean preference pair, log_ratio_chosen sits above log_ratio_rejected (the policy is learning to prefer chosen), and z is positive but bounded by the magnitudes the two log-probs can plausibly reach.

The divergence happens on pathological pairs. If chosen and rejected are textually near-identical, the model has no way to anchor confidence semantically, and gradients can push the log-probs in opposite directions without bound as training progresses. If the label is actually flipped (chosen is rated as preferred but is in fact worse), the optimizer pushes the policy toward a confidence that contradicts the reference, and the log-ratio difference can grow even faster.

As z grows, the sigmoid saturates. sigma(z) approaches 1 for large positive z, so log sigma approaches 0 and the loss approaches 0; for large negative z, sigma approaches 0, log sigma approaches negative infinity, and the loss explodes. The gradient of the loss with respect to the model parameters carries the same magnitudes. A single batch where z hits +/-20 produces a gradient that swamps every prior update.

Beta as the amplifier and fp16 as the trigger

Three fixes in cheapest to most invasive order

The deeper fix: audit the preference dataset

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Symptom timing	Likely cause	First fix
Flat from step 0	Structural break (LR, wrapping, labels)	Trace optimization path
Clean start, NaN mid-run (this case)	DPO logit divergence + fp16	Lower beta, clip, bf16
Loss drops then plateaus	Capacity or convergence	Check eval; bump rank if eval flat
Train down, eval up	Overfitting	Early stopping; reduce capacity
Oscillating loss	LR too high or small batch	Lower LR, larger batch, more warmup

Real products, models, and research that use this idea.

TRL DPOTrainer in 2026 defaults to bf16 on Hopper hardware specifically to avoid the fp16 NaN failure mode this question describes.
Llama 4 Maverick preference-tuning recipes ship with beta=0.05 by default after community reports of NaN spikes at beta=0.1 on noisy crowd-sourced pairs.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does NaN propagate forward indefinitely once it appears in DPO?

AdamW's first and second moment buffers store running averages that get multiplied by the gradient. A NaN gradient corrupts both moments, and from that step forward every update is NaN regardless of fresh gradients. The only recoveries are to checkpoint-restart from before the spike or to manually reset the optimizer state.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Blaming the learning rate when DPO spikes to NaN. The DPO loss has its own amplifier (beta) on a term that can diverge independently of LR, and the right diagnostic is the logit blow-up, not a generic optimization issue.

60 second bullets to scan on the way to the call.

The DPO loss form and what the logit term inside the sigmoid represents
Why beta amplifies the dangerous exponent and how it interacts with fp16

Primary sources. Browse if you want the original framing.