What training benefit does a long cosine tail provide near run end?

The **learning rate** sets how big each weight update is. A cosine schedule starts it high and lowers it along half of a cosine curve, from the peak to a small floor at the last step. The curve is flat at both ends. So the last part of the run, the tail, spends many steps at a small and slowly shrinking learning rate. The tail gives three benefits. First, the loss drops. Each update carries some noise, because it comes from a random batch of data. A big learning rate turns that noise into big jumps, so the model bounces around near a good solution instead of settling into it. As the learning rate shrinks, the jumps shrink and the loss falls. Second, the model keeps learning. The decay is slow, so small improvements still add up instead of stopping all at once. Third, the end of the run is calm. Checkpoints near the end differ very little, so the final model is stable and does not depend on the exact step where it was saved.

A long cosine tail gives the run a slow, smooth landing. The learning rate follows: $$\eta_t = \eta_{\min} + \tfrac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\frac{\pi t}{T}\right)$$ The curve is flat near t = T, so the last 20% of steps all run below about 10% of the peak-to-floor range. The first benefit is a lower loss. At a high learning rate, gradient noise keeps the weights bouncing across the walls of the loss valley. So the loss sits above what the model has learned. As the learning rate falls, the bouncing shrinks and the loss drops. Much of the final gain comes in this phase. The second benefit is **consolidation**. The decay is gradual, so the model still makes small useful updates while the noise falls, instead of freezing at one abrupt drop. The third benefit is stability. Late checkpoints differ little, so the loss curve is smooth and the final checkpoint does not depend on luck. The tail only works if the cycle length T matches the run. Chinchilla found that a cycle more than 25% longer than the run gave clearly worse models, because training ended before the decay finished.