Spot the errors in this KV-eviction strategy for long-running generation

When you generate text for a very long time, the **KV cache** keeps growing. Eventually it exceeds memory or the model's trained context length, and you have to drop something. The tempting fix is a sliding window: always evict the oldest tokens to make room for new ones. The passage states three confident claims, and all three are wrong. It says naive eviction degrades quality only slightly. It says attention focuses on recent tokens. It says the first tokens do not matter. In reality, quality collapses, attention does not focus only on recent tokens, and the first tokens matter enormously. The reason is a surprising effect called the **attention sink**. Softmax forces every token's attention weights to sum to one, even when a token has nothing useful to attend to. Models learn to dump that leftover weight onto the first few tokens. Those positions become a kind of pressure-release valve. Drop the first tokens and the valve disappears. The softmax math destabilizes and output turns incoherent within a handful of steps. StreamingLLM fixes this cheaply: keep the first four tokens pinned forever, then slide a window over everything after them.

**The attention sink** is the key idea this passage misses. Softmax forces each token's attention to sum to one. When a token has little real signal to gather, the model still must place that mass somewhere. Transformers learn to dump the surplus onto the first few positions. Those early tokens become high-attention sinks regardless of their content. This is an emergent property, not something anyone designed. So naive sliding-window eviction is a trap. The moment the oldest block leaves the cache, the sink tokens are evicted. Attention loses its dumping ground. Perplexity spikes and the output collapses within a few steps, not gradually. The claim of a small monotonic degradation is exactly backwards. The second error follows. Attention does not focus only on recent tokens. The first positions carry disproportionate weight even at long range. The third error is the same mistake again: those tokens have unusually high softmax scores, not low ones. StreamingLLM is the fix. Keep a few sink tokens pinned permanently. Slide a fixed window over the recent context. This lets a model generate over millions of tokens with stable perplexity, without any retraining and without growing the cache.