Why does naive sliding-window KV eviction break and how does StreamingLLM's attention-sink fix it?

**Attention** uses a softmax, which forces the attention weights over all past tokens to add up to one. The model cannot say "attend to nothing this step." Even when no past token is relevant, that probability mass has to land somewhere. During training, models discover a cheap trick. They dump this leftover mass onto the very first few tokens of the sequence, often the start token. Those positions become stable parking spots called attention sinks. They carry little meaning but absorb the excess so the real attention stays clean. To run past the trained context length, a naive scheme uses a sliding window. It keeps only the most recent tokens in the KV cache and drops the oldest ones. The problem is that the oldest tokens include those first sink positions. Once the sinks are evicted, the leftover softmax mass has nowhere familiar to go. It spreads across middle tokens, amplifying noise. Output becomes incoherent within a handful of tokens. The **StreamingLLM** fix is structural and tiny. Always keep the first few KV entries pinned, then slide the window over everything after them. The model keeps its sinks, and generation stays coherent over arbitrarily long streams at constant memory cost.

Softmax attention must sum to one over the visible tokens. The model cannot output a true "attend to nothing" distribution. So it learns to offload surplus probability onto a few fixed early positions. These positions become **attention sinks**. Often the start token, they are nearly content free yet structurally load bearing. They soak up mass that would otherwise distort the real attention pattern. Naive sliding-window eviction drops the oldest KV blocks to cap memory. Eventually it evicts those first sink tokens. The surplus mass then has nowhere stable to land. It redistributes across mid-context positions with high gain on noise. Quality collapses within a few tokens; the text turns incoherent fast. StreamingLLM's fix is structural, not statistical. It pins the first few KV entries permanently. These are the sinks. It then applies the sliding window only to the tokens after them. The model keeps the parking spots it was trained to use. Generation stays coherent over effectively unbounded streams. The cost is a fixed constant of roughly four blocks. The benefit is stable quality on long agent loops and persistent assistants without any retraining.