Gradient norm clipping, name the failure mode it exists to prevent

Fine-tuning runs see thousands of gradient steps. Most are well behaved. A handful are not, and that small handful is what gradient clipping defends against. The failure mode is the **exploding-gradient step**. A single batch with an unusual property, a very long sequence, a mislabelled example, an out of distribution prompt, can produce a gradient whose L2 norm is many times larger than the running average. The optimizer multiplies that giant gradient by the learning rate and steps. The weights land far outside the well-behaved region they were sitting in. The symptom in the training log is a sudden loss spike followed by a flat, high loss that never recovers. The model fell out of the basin and the rest of the run is wasted compute. Gradient clipping caps the global L2 norm of the gradient before the step. If the norm is at or below a threshold, commonly 1.0, nothing changes. If it exceeds the threshold, every component is scaled down by the same factor so the new norm equals the threshold. The direction of the gradient is unchanged. Only its magnitude is capped.

**Gradient norm clipping** prevents one bad batch from destroying the run. During fine-tuning the gradient magnitude is usually well behaved, but tail events happen: an unusually long sequence, a noisy label, a chat-template bug that mis-masks. The resulting gradient L2 norm can be ten to a hundred times the running average. Multiplied by the learning rate, that step pushes weights well outside the basin they were sitting in. The loss spikes and rarely recovers. The fix is simple. Compute the L2 norm across the whole parameter vector, and if it exceeds a threshold, rescale every component by: $$g \leftarrow g \cdot \min\!\left(1,\ \frac{\tau}{\lVert g \rVert_2}\right)$$ The direction of g is preserved. Only its length is capped at tau. A typical threshold is `max_grad_norm = 1.0` for transformer SFT. Two subtleties matter. First, the clip is global, computed once across all trainable parameters, so layers keep their relative scale. Second, clipping should run before the optimizer step but after any gradient accumulation, so the buffer is the gradient the optimizer will actually use.