Same topic, related formats. Practice these next.
When generation must go past the model's trained context length, a naive sliding-window eviction (drop the oldest KV blocks) catastrophically breaks output quality. Explain WHY, and how StreamingLLM's 'attention sink' design fixes the failure.
Softmax forces attention to sum to one, so models park excess mass on the first few tokens. Evict those sinks and quality collapses; StreamingLLM pins them.
Imagine a room where everyone must always cast votes totaling exactly 100 percent, even when no real candidate deserves them. People learn to park their leftover votes on one bored person in the front row who never minds. That person is the attention sink. Now suppose you run out of chairs and start removing people from the front of the room to make space. The moment you remove that front-row vote-parker, everyone's leftover votes have nowhere safe to go. They scatter onto random people in the middle, who suddenly seem far more important than they really are. The whole count turns to nonsense. StreamingLLM's fix is simple: never remove those front-row people, no matter how crowded the back of the room gets.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: softmax must sum to one, then emergent sinks, then why eviction cliffs quality, then StreamingLLM pin plus slide, then the streaming stability versus recall distinction and constant-memory cost.
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming the oldest tokens are safe to drop because they are far from the current position. The first few tokens carry no semantic value yet are load bearing as attention sinks.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.