Click any words you think contain an error. Click again to unmark.
Clipping bounds damage but does not remove a cause; a 40% clip rate is itself an instability signal that demands a data and logit audit.
If your kitchen smoke alarm keeps going off, taping a cloth over it makes the noise stop. The fire is still there. Gradient clipping at a fixed threshold is the cloth: it muffles the worst gradients so the loss curve looks calm, but whatever was producing those huge gradients (a corrupted batch, attention logits drifting upward, a numerical precision issue) is still happening inside the model. And if the alarm muffler is triggering on almost every meal, you should not be writing a memo saying the kitchen is safe. You should be looking for the actual fire.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Gradient clipping is one of the most useful and most over-trusted tools in pretraining. It works, in the narrow sense that it prevents single bad updates from wrecking the run. It does not work, in the broader sense that it never tells you why the bad update happened. The memo in this question is a real pattern: clipping makes the loss curve look calm, the team declares the incident closed, and the underlying mechanism keeps growing until the next, worse, event.
This deep dive walks through what clipping actually does to a step, why clip rate is the metric you should be watching, what biases aggressive clipping introduces into the optimizer, and what the mechanism-level fixes look like.
What clipping does to a single step
Global gradient norm clipping rescales the full gradient vector when its L2 norm exceeds a threshold:
Notice three properties. First, it only ever shrinks; there is no upward correction. Second, it scales the entire vector uniformly, so the direction is preserved exactly. Third, it acts after backprop and before the optimizer step, so the optimizer's moment estimates are built from the clipped gradient, not the raw one.
Those properties make clipping a clean safety net when it fires rarely. The bias it introduces is small because the rare clipped steps are a tiny fraction of the moment EMAs. They become large when clipping fires often: the moments now reflect a systematically truncated distribution of gradients, the optimizer's update scale is no longer calibrated, and the effective learning rate on directions that frequently exceed the threshold is silently reduced.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The PaLM technical report documents that rerunning from a pre-spike checkpoint with skipped data shards (not just clipping harder) was the recipe that actually stabilized training
- OLMo 2 published z-loss and qk-layernorm as targeted mitigations for growing attention logits, framing them explicitly as mechanism-level fixes that clipping cannot replace
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you set the clip threshold on a new architecture you have no prior runs for?
Run a short calibration job, measure the gradient-norm distribution, set the threshold around the 95 to 99 percent quantile of the stable region, and treat any sustained rise in clip rate as a signal to retune rather than as evidence the threshold is wrong.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Declaring an instability fixed because clipping made the loss curve look smooth, while clip rate climbs to 30 percent or more and the underlying mechanism keeps growing.
60 second bullets to scan on the way to the call.
What does gradient clipping actually do to a step where the norm exceeds the threshold?
Why is a high clip rate a problem rather than a sign that clipping is working?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.