Click any words you think contain an error. Click again to unmark.
Naive sliding-window eviction drops the first tokens, which are attention sinks, so quality collapses. StreamingLLM fixes it by pinning a few sink tokens plus the recent window.
Imagine a long meeting where everyone keeps glancing at the chairperson before they speak, almost out of habit, even when the chair is not the topic. That quick glance steadies them. Now imagine the chair quietly leaves. Suddenly everyone's instinct misfires, glances land on random people, and the conversation falls apart. A transformer behaves the same way. The very first tokens act like that chairperson: the model learned to dump leftover attention onto them so the math stays balanced. If you evict those first tokens to save memory, the balance breaks and the output turns to garbage. The fix is simple. Keep the first few tokens forever, slide a window over the rest, and the meeting keeps running smoothly.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The attention sink is a counterintuitive finding about trained LLMs, and this passage gets it backwards three times. The problem is what to do when an agent loop or chat session runs past the model's trained context length and the KV cache can no longer hold every token.
The obvious answer, and the one the passage endorses, is a naive sliding window: at each step, evict the oldest KV block to make room for the new token. The passage then justifies this with two confident-sounding claims: attention focuses on recent tokens, and the first tokens carry negligible attention, so dropping them is harmless. Both claims feel right and both are wrong.
The earliest tokens become attention sinks, positions onto which the model dumps surplus softmax mass. Evict them and the model does not degrade gracefully; it collapses. StreamingLLM fixes this by changing only which tokens the cache keeps.
Why softmax normalization creates attention sinks
Every attention head computes a softmax over the keys, and softmax outputs always sum to one. That constraint is the root cause. A head must distribute its full unit of attention mass across the available tokens even when none of them are genuinely useful for the current query.
There is no built-in option to attend to nothing. Faced with this, models learn a workaround during training. They route the leftover, uninformative mass to a fixed, always-available position rather than smearing it as noise across meaningful tokens.
The first tokens are the natural target. Under causal masking, position zero is visible to every later query in the sequence, so it is the one location every head can reliably reach. The model puts surplus mass there. The result is an emergent attention sink: a position with very high softmax weight but very little semantic role. Nobody designed this; it results from the normalization constraint plus gradient descent.
Causal masking explains why the early positions, not late ones, take this role. A query at position 5000 can attend to any of positions 0 through 5000. Position 4999 is visible only to queries at 4999 and later, and in a shorter sequence it does not exist at all. Position 0 is the only key that has existed since the very first decode step and is guaranteed present in every causal context the model ever sees. Gradient descent rewards consistency, so the network settles on the one key every query can see. The first token becomes the shared place for excess probability. The StreamingLLM paper finds this focus on initial tokens in all layers and heads beyond the bottom two layers.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- StreamingLLM, from MIT and Meta, demonstrated stable perplexity up to four million tokens by pinning four sink tokens plus a recent window.
- GPT-OSS-120B gives each attention head a learned bias in the softmax denominator, which acts as an attention sink, and alternates 128-token banded window layers with full attention layers.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does softmax normalization create attention sinks in the first place?
Trace the constraint that every attention row sums to one. A head that finds nothing useful nearby still must place its full mass somewhere, so the model learns to put surplus weight on a fixed, always-present position rather than smearing noise across real tokens.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming attention concentrates on recent tokens, so the oldest KV is safe to drop. The first few tokens are sinks, and evicting them collapses quality abruptly.
60 second bullets to scan on the way to the call.
Why softmax normalization forces every attention row to allocate its full mass
What an attention sink is and which token positions become sinks
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.