Click any words you think contain an error. Click again to unmark.
Naive sliding-window eviction drops the first tokens, which are attention sinks, so quality collapses. StreamingLLM fixes it by pinning a few sink tokens plus the recent window.
Imagine a long meeting where everyone keeps glancing at the chairperson before they speak, almost out of habit, even when the chair is not the topic. That quick glance steadies them. Now imagine the chair quietly leaves. Suddenly everyone's instinct misfires, glances land on random people, and the conversation falls apart. A transformer behaves the same way. The very first tokens act like that chairperson: the model learned to dump leftover attention onto them so the math stays balanced. If you evict those first tokens to save memory, the balance breaks and the output turns to garbage. The fix is simple. Keep the first few tokens forever, slide a window over the rest, and the meeting keeps running smoothly.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The attention sink is one of the most counterintuitive findings in modern LLM serving, and this passage gets it exactly backwards three times in a row. The question is what to do when an agent loop or chat session runs past the model's trained context length and the KV cache can no longer hold every token.
The obvious answer, and the one the passage endorses, is a naive sliding window: at each step, evict the oldest KV block to make room for the new token. The passage then justifies this with two confident-sounding claims: attention focuses on recent tokens, and the first tokens carry negligible attention, so dropping them is harmless. Both claims feel right and are both wrong.
The truth is that the earliest tokens become attention sinks, positions onto which the model dumps surplus softmax mass. Evict them and the model does not degrade gracefully; it collapses. This deep dive explains why the sink effect emerges from softmax normalization, why naive eviction fails as a cliff rather than a slope, and how StreamingLLM fixes it with a recipe so cheap it almost feels like a trick.
Why softmax normalization manufactures attention sinks
Every attention head computes a softmax over the keys, and softmax outputs always sum to one. That constraint is the root cause. A head must distribute its full unit of attention mass across the available tokens even when none of them are genuinely useful for the current query.
There is no built-in option to attend to nothing. Faced with this, models learn an elegant hack during training. They route the leftover, uninformative mass to a fixed, always-available position rather than smearing it as noise across meaningful tokens.
The first tokens are the natural target. Under causal masking, position zero is visible to every later query in the sequence, so it is the one location every head can reliably reach. The model parks surplus mass there. The result is an emergent attention sink: a position with very high softmax weight but very little semantic role. Nobody designed this; it falls out of the normalization constraint plus gradient descent.
It helps to see why the early positions, not late ones, win this role. A query at position 5000 can attend to any of positions 0 through 5000. Position 4999 is recent but volatile: it appears in only a handful of queries before it too becomes old. Position 0 is the only key that has existed since the very first decode step and is guaranteed present in every causal context the model ever sees. Gradient descent rewards consistency, so the network converges on the most stable available anchor. The first token becomes the shared off-ramp for excess probability, and the effect strengthens with depth as later layers inherit and amplify the pattern.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- StreamingLLM, from MIT and Meta, demonstrated stable perplexity over four million tokens by pinning four sink tokens plus a recent window.
- Llama 4 and Mistral Large 3 ship with attention-sink aware streaming so agent loops stay coherent far past the trained context length.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does softmax normalization create attention sinks in the first place?
Trace the constraint that every attention row sums to one. A head that finds nothing useful nearby still must place its full mass somewhere, so the model learns to park surplus weight on a fixed, always-present position rather than smearing noise across real tokens.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming attention concentrates on recent tokens, so the oldest KV is safe to drop. The first few tokens are sinks, and evicting them collapses quality abruptly.
60 second bullets to scan on the way to the call.
Why softmax normalization forces every attention row to allocate its full mass
What an attention sink is and which token positions become sinks
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.