When generation must go past the model's trained context length, a naive sliding window eviction (drop the oldest KV blocks) catastrophically breaks output quality. Explain WHY, and how StreamingLLM's 'attention sink' design fixes the failure.
Softmax forces attention to sum to one, so models park excess mass on the first few tokens. Evict those sinks and quality collapses; StreamingLLM pins them.
Imagine a room where everyone must always cast votes totaling exactly 100 percent, even when no real candidate deserves them. People learn to park their leftover votes on one bored person in the front row who never minds. That person is the attention sink. Now suppose you run out of chairs and start removing people from the front of the room to make space. The moment you remove that front-row vote-parker, everyone's leftover votes have nowhere safe to go. They scatter onto random people in the middle, who suddenly seem far more important than they really are. The whole count turns to nonsense. StreamingLLM's fix is simple: never remove those front-row people, no matter how crowded the back of the room gets.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
StreamingLLM sits at the intersection of two interview favorites: how softmax attention actually behaves, and how production serving stacks cap the KV cache. The question is hard because the failure it describes is counterintuitive. Dropping the oldest, most distant tokens feels obviously safe. They are far from the current decode position and seem semantically stale. Yet doing so detonates output quality almost instantly.
The resolution hinges on a subtle property of softmax normalization and an emergent behavior that pretrained transformers reliably learn. The first few tokens of a sequence quietly become load bearing, not because of what they say, but because of the role they play in the attention arithmetic. Evict them and the whole distribution destabilizes.
This deep dive walks through the softmax constraint that creates attention sinks, why naive sliding-window eviction triggers a sudden quality cliff rather than a gentle decay, exactly how StreamingLLM's pin plus slide layout repairs it, the position-encoding detail that makes it work, and the crucial limitation that separates streaming stability from genuine long-range recall.
The softmax constraint that births attention sinks
Self-attention scores every visible key against the current query, then passes the scores through a softmax. The softmax normalizes its outputs to sum to exactly one across all visible positions:
There is no way to express "attend to nothing this step." Every token receives some nonzero weight, however small. This creates a problem for any decode step where no past token is genuinely relevant. The head still has a full unit of probability to distribute, and it cannot abstain. A head that has already gathered the information it needs has no clean way to switch itself off; the normalization forces it to keep voting.
During pretraining, the model discovers a cheap and stable solution: route that surplus probability onto a few fixed early positions. Those positions, typically the start token and the first one to four tokens, become attention sinks. They are nearly content free, yet they reliably soak up the excess that would otherwise distort the real attention pattern. The behavior is emergent and consistent across heads and layers. It is a structural artifact of the normalization, not a property of what those first words mean.
Why the first tokens specifically? Two reasons reinforce each other. They are visible to every later query under causal masking, so they are the one set of keys guaranteed to be present at every step of every sequence. And because they appear in essentially every training example at the same relative location, gradient descent can reliably learn to over-weight them without conflicting signals. The result is a value vector at those positions that contributes little to the output, paired with key vectors that consistently win a slice of the softmax. The mass parked there is effectively discarded, which is exactly what the model wants when no real token deserves attention.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- StreamingLLM (MIT and Meta, 2023) demonstrated stable multi-million token decoding by pinning four sink tokens plus a recent window.
- vLLM and SGLang expose sink-aware sliding-window KV configs for long-running agent sessions in 2026 deployments.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the surplus attention mass have to go somewhere instead of just being zero?
Trace the softmax denominator. It normalizes the exponentiated scores to sum to one over visible keys. There is no abstain option, so even tiny logits get nonzero weight; the model learns to route that residue to stable early keys it can safely ignore.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the oldest tokens are safe to drop because they are far from the current position. The first few tokens carry no semantic value yet are load bearing as attention sinks.
60 second bullets to scan on the way to the call.
Why softmax normalization forces surplus attention mass onto some position
How the first few tokens become emergent attention sinks during pretraining
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.