Click any words you think contain an error. Click again to unmark.
Clipping bounds damage but does not remove a cause; a 40% clip rate is itself an instability signal that demands a data and logit audit.
If your kitchen smoke alarm keeps going off, taping a cloth over it makes the noise stop. The fire is still there. Gradient clipping at a fixed threshold is the cloth: it muffles the worst gradients so the loss curve looks calm. But whatever was producing those huge gradients (a corrupted batch, attention logits drifting upward, a numerical precision issue) is still happening inside the model. And if the alarm muffler is triggering on nearly half of all meals, you should not be writing a memo saying the kitchen is safe. You should be looking for the actual fire.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Gradient clipping is one of the most useful and most over-trusted tools in pretraining. It works, in the narrow sense that it prevents single bad updates from wrecking the run. It does not work, in the broader sense that it never tells you why the bad update happened. A common failure pattern follows: clipping makes the loss curve look calm, the team declares the incident closed, and the underlying mechanism keeps growing until the next, worse, event.
What clipping does to a single step
Global gradient norm clipping rescales the full gradient vector when its L2 norm exceeds a threshold:
Clipping has three properties. First, it only ever shrinks; there is no upward correction. Second, it scales the entire vector uniformly, so the direction of that step is preserved exactly. Third, it acts after backprop and before the optimizer step, so the optimizer's moment estimates are built from the clipped gradient, not the raw one.
Those properties make clipping a clean safety net when it fires rarely. The bias it introduces is small because the rare clipped steps are a tiny fraction of the moment EMAs. The bias becomes large when clipping fires often. The moments now reflect a systematically truncated distribution of gradients, and the optimizer's update scale is no longer calibrated. Directions that often push the gradient past the threshold get less weight across steps than they should. Each step keeps its direction, but the average over many steps is biased.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The PaLM technical report handled loss spikes by restarting from a checkpoint about 100 steps before the spike and skipping roughly 200 to 500 data batches, rather than by clipping harder
- OLMo 2 adopted qk-layernorm (to bound attention logits) and z-loss (to keep output logits in check) as targeted stability fixes in its training recipe
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you set the clip threshold on a new architecture you have no prior runs for?
Run a short calibration job and measure the gradient-norm distribution. Set the threshold around the 95 to 99 percent quantile of the stable region. Treat any sustained rise in clip rate as a signal to retune rather than as evidence the threshold is wrong.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Declaring an instability fixed because clipping made the loss curve look smooth, while clip rate climbs to 30 percent or more and the underlying mechanism keeps growing.
60 second bullets to scan on the way to the call.
What does gradient clipping do to a step where the norm exceeds the threshold?
Why is a high clip rate a problem rather than a sign that clipping is working?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.