Why not just retrain models to eliminate attention sinks entirely?

The teammate's plan does not work because **sinks are not a bug that retraining can patch**. They are an emergent consequence of one of softmax's core constraints: every row of the attention weight matrix must sum to exactly 1, even when no key really matches the current query. During training, the model has to put that mandatory probability mass somewhere. The first token (often BOS) is convenient because it is content-light, predictable, and present in every sequence the model sees. So the model learns to dump excess attention weight there. That is the sink. If you retrain after removing the BOS sink, the model still has to satisfy softmax-sum-to-1. It picks the next most convenient token and promotes it to sink duty. Often that is a real content token, whose representation now gets distorted because attention heads are routing meaningless no-op weight through it. **The actual fix is register tokens.** Add a small number of dedicated tokens at the start of every sequence whose explicit purpose is to absorb sink weight. Models trained this way (Darcet et al. for ViT, similar tricks for LLMs) show smoother attention maps, fewer outlier activations, and better long-context behavior because real content tokens are no longer doing sink duty. StreamingLLM exploits the same insight at inference time by keeping the first few tokens permanently in the window so they remain available as sinks.

**Sinks emerge from softmax-must-sum-to-1, so retraining without the current sink does not eliminate them; it just relocates the sink to the next-best token.** The mechanism is structural. For every query position, softmax produces a row that sums to exactly 1 over all keys. When no key in the context is a good content match (which is most queries for most heads, most of the time), the model still has to put that mandatory probability mass somewhere. The first token, usually BOS, is the natural dump position because it is content-light, present in every sequence, and at a predictable index. The model learns during pretraining to route excess attention there. That is what we call a sink. If you retrain without BOS or penalize first-position attention, the constraint does not go away. The model finds a new sink. Often it is a real content token whose representation then gets distorted by carrying both content signal and sink load, hurting downstream tasks. **The real fix: give the model dedicated no-op tokens.** Register tokens (introduced for ViT by Darcet et al., now used in LLM training too) are a handful of learned tokens prepended to every sequence whose explicit purpose is to soak up sink weight. Models trained with registers show three robust improvements: smoother and more interpretable attention maps, far fewer outlier activations in the residual stream, and better long-context behavior because content tokens no longer carry the sink load. StreamingLLM gives the same insight at inference time: instead of changing training, it keeps the first few tokens permanently in the KV cache window so the sinks remain available regardless of how far decoding has progressed. Without that, sliding-window inference catastrophically degrades once the sinks fall out of scope.