Spot the errors in this KV-eviction strategy for long-running generation

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Naive sliding-window eviction drops the first tokens, which are attention sinks, so quality collapses. StreamingLLM fixes it by pinning a few sink tokens plus the recent window.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a long meeting where everyone keeps glancing at the chairperson before they speak, almost out of habit, even when the chair is not the topic. That quick glance steadies them. Now imagine the chair quietly leaves. Suddenly everyone's instinct misfires, glances land on random people, and the conversation falls apart. A transformer behaves the same way. The very first tokens act like that chairperson: the model learned to dump leftover attention onto them so the math stays balanced. If you evict those first tokens to save memory, the balance breaks and the output turns to garbage. The fix is simple. Keep the first few tokens forever, slide a window over the rest, and the meeting keeps running smoothly.

Key concepts

Attention sink StreamingLLM KV cache eviction Sliding-window attention

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: softmax must sum to one creating sinks on early tokens, why naive eviction collapses abruptly, the StreamingLLM pin sinks plus window fix, and the streaming vs context extension distinction.

Real-world examples

Real products, models, and research that use this idea.

StreamingLLM, from MIT and Meta, demonstrated stable perplexity over four million tokens by pinning four sink tokens plus a recent window.
Llama 4 and Mistral Large 3 ship with attention-sink aware streaming so agent loops stay coherent far past the trained context length.
vLLM and SGLang both expose sink-token retention in their KV cache eviction policies for long-running production serving.
GPT-5.5 and Claude Opus 4.7 use sink-aware streaming internally so multi-hour agent sessions do not degrade as the cache window slides.
DeepSeek V4 documents a dedicated learnable sink token so real content tokens are not co-opted as the softmax dumping ground.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does softmax normalization create attention sinks in the first place?

Sign in or go Pro to see the approach for this follow-up.

QWhy are the FIRST tokens chosen as sinks rather than some middle position?

Sign in or go Pro to see the approach for this follow-up.

QHow does a dedicated learnable sink token improve on pinning the first real tokens?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming attention concentrates on recent tokens, so the oldest KV is safe to drop. The first few tokens are sinks, and evicting them collapses quality abruptly.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.