Describe the attention sink: why token 0 hoards attention mass

**The attention sink** is an emergent behavior of trained transformers: large amounts of attention mass concentrate on the first few token positions, regardless of their actual content. The cause is softmax. Each head's attention weights must sum to 1.0 over all positions visible to the current query. When the head has nothing useful to attend to, the mass still has to go somewhere. Models learn to dump it on token 0, which is always visible to every causal query, making it a stable parking spot. Why it matters: sink tokens become load-bearing. The model has learned that they hold the no-op mass. If you remove them from the KV cache (which naive sliding windows do for memory reasons), the softmax distribution shifts across the board. Output quality collapses immediately. The fix in production: keep a small number of sink tokens (typically 4) in the cache permanently, even when the rest slides. This is the StreamingLLM recipe. Some 2026 models train a dedicated learnable sink token to make the mechanism explicit and robust.

**The attention sink** is the emergent observation that trained transformers route disproportionate attention mass to the first few tokens, with token 0 the dominant target. The cause is structural, not data dependent. Softmax constraint. Each attention head outputs a probability distribution that sums to 1. When no key is genuinely relevant to the query, the probability mass cannot just vanish; it has to land somewhere. Under causal masking, the earliest tokens are visible to every position, so they are the convenient default sink. $$\sum_j \alpha_{ij} = 1 \quad \text{for every query } i$$ Production consequence. Naive long-context inference uses a sliding KV cache to bound memory. Evict the early tokens and the sink mass has nowhere to go; the model redistributes it across the remaining keys, distorting every per-head function. Quality collapses within a few hundred tokens past the eviction. Fixes. StreamingLLM pins the first 4 tokens in the cache permanently, sliding only the middle. Mistral and several 2026 servers ship the same approach. Anthropic and others have explored an explicit learnable sink token trained from scratch, which dedicates one position to the role. Sigmoid attention and unnormalized variants remove the unit-mass pressure entirely, but those are research moves rather than production defaults.