What's the 'attention sink' phenomenon, and how does StreamingLLM exploit it?

**Attention sinks** are a structural feature of trained autoregressive LLMs. For almost every query at almost every layer, a disproportionate fraction of attention mass lands on the first few tokens (often just BOS), even when those positions are semantically unrelated to the current context. Why? Causal attention softmax distributes mass that sums to 1. There is no don't-attend option. When no past token is strongly relevant, the model dumps leftover mass somewhere innocuous; the first tokens are perfect (always present, low semantic interference). Gradient descent locks this in during pretraining. This matters for streaming. The naive approach is sliding window: keep only the K most recent tokens, drop older ones. Once the sequence exceeds K, the sinks fall off. Softmax still must allocate 100% somewhere, but now lands on positions never trained as overflow valves. Output degenerates: incoherence, repetition, collapse, within roughly one window length. **StreamingLLM's fix**: always keep the first 4 tokens in cache plus a sliding window of the K most recent. Sinks remain available, softmax keeps working as trained, and generation stays coherent. Enables 100k+ token streaming on models trained at 2-4k. Sinks must retain their original positional encoding (RoPE rotational phase); renumbering breaks the overflow valve.

Across trained autoregressive LLMs, the first 4 token positions absorb roughly 20-50% of attention mass per head regardless of their content. Xiao 2023 named these positions **attention sinks**. They emerge because causal softmax must distribute attention mass that sums to one; when no past token is strongly relevant, the model dumps the excess on the always present first tokens. The behavior emerges robustly across architectures (Llama, MPT, Pythia, GPT-2) because gradient descent during pretraining finds this overflow valve solution. Naive sliding window streaming evicts those sinks once the sequence exceeds the window. Softmax loses its overflow valve, distributions become pathological, and generation collapses into incoherence within a window length. **StreamingLLM's fix** is to keep the first 4 tokens permanently and pair them with a sliding window of recent W tokens, dropping everything in between. The sinks must retain their original positional encoding to preserve the overflow valve property. The change is inference time only with no retraining, and enables 100k+ token streaming on models trained at 2-4k. Production stacks (vLLM, SGLang, TRT-LLM) implement sink aware KV management for long stream chat sessions in 2026.