Explain the 'attention sink' phenomenon discovered by Xiao et al. 2023 (StreamingLLM). What is the phenomenon, why does it exist, and how does StreamingLLM exploit it to enable effectively unlimited length streaming generation?
Free · 2 AI evals / day
**Attention sinks** are a structural feature of trained autoregressive LLMs. For almost every query at almost every layer, a disproportionate fraction of attention mass lands on the first few tokens (often just BOS), even when those positions are semantically unrelated to the current context.
Why? Causal attention softmax distributes mass that sums to 1. There is no don't-attend option. When no past token is strongly relevant, the model dumps leftover mass somewhere innocuous; the first tokens are perfect (always present, low semantic interference). Gradient descent locks this in during pretraining.
This matters for streaming. The naive approach is sliding window: keep only the K most recent tokens, drop older ones. Once the sequence exceeds K, the sinks fall off. Softmax still must allocate 100% somewhere, but now lands on positions never trained as overflow valves. Output degenerates: incoherence, repetition, collapse, within roughly one window length.
**StreamingLLM's fix**: always keep the first 4 tokens in cache plus a sliding window of the K most recent. Sinks remain available, softmax keeps working as trained, and generation stays coherent. Enables 100k+ token streaming on models trained at 2-4k.
Sinks must retain their original positional encoding (RoPE rotational phase); renumbering breaks the overflow valve.
Across trained autoregressive LLMs, the first 4 token positions absorb roughly 20-50% of attention mass per head regardless of their content. Xiao 2023 named these positions **attention sinks**.
They emerge because causal softmax must distribute attention mass that sums to one; when no past token is strongly relevant, the model dumps the excess on the always present first tokens. The behavior emerges robustly across architectures (Llama, MPT, Pythia, GPT-2) because gradient descent during pretraining finds this overflow valve solution.
Naive sliding window streaming evicts those sinks once the sequence exceeds the window. Softmax loses its overflow valve, distributions become pathological, and generation collapses into incoherence within a window length.
**StreamingLLM's fix** is to keep the first 4 tokens permanently and pair them with a sliding window of recent W tokens, dropping everything in between. The sinks must retain their original positional encoding to preserve the overflow valve property. The change is inference time only with no retraining, and enables 100k+ token streaming on models trained at 2-4k. Production stacks (vLLM, SGLang, TRT-LLM) implement sink aware KV management for long stream chat sessions in 2026.
Quick reference
TL;DR
Trained LLMs dump excess softmax mass onto the first few tokens. StreamingLLM keeps those sinks plus a recent window, unlocking 100K+ token streaming.
Imagine a party host who has to hand out exactly 100 percent of their attention to the room at every moment, no exceptions. When nothing important is going on, the host has to look somewhere, so they default to glancing at the first guests who walked in. Those first arrivals quietly soak up the leftover gaze. Now make the host only look at the most recent guests and ignore those early arrivals. The leftover attention has nowhere to go, the host gets flustered, and the conversation falls apart within minutes. StreamingLLM says: no matter how long the party runs, always let the host keep a corner of an eye on those first few guests.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Use cases where the full original context must remain accessible mid-stream StreamingLLM drops everything between the sinks and the recent window; semantically important middle tokens are gone and cannot be re-attended.
Models pretrained with very long native context where rolling window eviction is unnecessary If the model handles 1M tokens natively (Gemini 3.1 Pro long context, some Llama 4 Maverick variants), keeping the full KV cache wins on quality; StreamingLLM is for serving beyond the trained range.
Random access workloads such as long document QA over the middle of a transcript StreamingLLM assumes a streaming generation pattern; arbitrary lookup into evicted middle tokens is not supported.
Architectures without softmax attention (state space models, linear attention variants) Attention sinks are a softmax phenomenon; Mamba style state space models and kernel feature map linear attention need a different long stream strategy.
Cases where positional encoding cannot be applied at each token's original position Sinks must retain their original RoPE phase; engine paths that re-index positions in the cache break the overflow valve mechanism.
Quick scan · 2–4 min
Everything important, quickly.
Walk the phenomenon and its softmax overflow valve explanation, the failure mode of naive sliding window, StreamingLLM's two part cache, the RoPE positional encoding constraint, the empirical context length gains, and the follow on research into explicit learned sinks.
Side-by-side
Approach
Cache content
Long context behavior
Notes
Naive sliding window (K tokens)
Last K tokens only
Collapses after first window
Sinks evicted → softmax has no overflow valve
StreamingLLM (4 sinks + K window)
First 4 + last K tokens
Coherent over arbitrarily long sequences
Sinks retained at original positions
Full long context training (e.g. Llama-3-128K)
Entire context
Native long context
Expensive to train; memory scales linearly
Position interpolation / YaRN
Entire context, rescaled positions
Extends trained range
Requires some fine-tuning; orthogonal to sinks
Real-world examples
Real products, models, and research that use this idea.
1StreamingLLM's GitHub release enabled Llama-2-7B to generate coherent text for millions of tokens, a viral demo at the time.
2Modern 2026 streaming serving stacks (vLLM, SGLang, TRT-LLM) implement sink aware KV management for very long stream chat sessions.
3Mistral Large 3's sliding window training accounts for this implicitly; the model is trained on sliding windows so the sinks as overflow valve mechanism adapts.
4Gemma 4, Llama 4 Maverick, and DeepSeek V4 are trained at long native context but still exhibit attention sinks empirically; the phenomenon is robust across architectures and scales.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do attention sinks interact with RoPE positional encoding?
Sign in or go Pro to see the approach for this follow-up.
QWhy don't attention sinks shift to later positions during training?
Sign in or go Pro to see the approach for this follow-up.
QWhat's a 'learned attention sink' and why might it outperform emergent sinks?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating attention sinks as a bug or curiosity rather than a critical structural feature; or trying to fix streaming by extending the window without preserving the sinks.