Name two production KV-cache eviction policies and what each prioritizes

Short answer·Medium·4.0 · 0·~3 min·Relevant atAnthropic NVIDIA

Attempt it

When generation pushes the KV cache past its memory budget at long context, the server must evict cache entries to keep going. Name two production-grade eviction policies, describe what each keeps and what each drops, and note the bookkeeping cost of each.

Free · 2 AI evals / day

When a long generation pushes the KV cache past available memory, the server must drop some cached K, V entries to keep generating. Two production policies dominate: **1. Sliding-window eviction**, keep the most recent W tokens; drop anything older. Bookkeeping is essentially free (a circular buffer or moving pointer). The cost: anything older than W is gone, so long-range references die. This is implicit in models trained with sliding-window attention like Mistral 7B and is the simplest fallback for stretched context. **2. Heavy-hitter eviction (H2O)**, score each cached token by how much attention weight it has received historically. Keep the top-k highest-scored tokens plus a recent local window; drop the rest. Bookkeeping requires per-token running scores updated each step. The win: better quality preservation at the same cache budget because semantically important tokens (entities, topic anchors) survive. Production often blends these. **StreamingLLM** keeps a few 'attention sink' tokens at the very start of the sequence (because of attention-sink phenomena where the first few positions absorb disproportionate weight, dropping them collapses the softmax) plus a recent window. The trade-off is quality vs bookkeeping cost: sliding-window is cheap, heavy-hitter is smarter, hybrids try to get both.

Two policies cover most production deployments. **Sliding-window eviction.** Keep only the last W tokens of K, V; drop the oldest as new tokens are added. Implementation is a circular buffer or a moving start-pointer, essentiallyzero bookkeeping cost. The quality cost is that anything beyond W is unreachable for the rest of the generation. Used implicitly by Mistral 7B's sliding-window attention and as the default fallback in vLLM and SGLang for context-bounded serving. **Heavy-hitter eviction (H2O style).** Track per-token attention weight accumulated over recent decode steps. At eviction time, keep the top-k highest-scored tokens plus a small local window of the most recent tokens; drop the rest. Bookkeeping is a per-token running score updated every step. The quality win is substantial: semantically important tokens (entities, topic anchors, structural markers) survive eviction even when they are far back in the sequence, while uninformative filler is dropped. **Hybrids in production.** **StreamingLLM** is the canonical example: keep the first few 'attention sink' tokens permanently (they absorb disproportionate softmax mass and dropping them destabilizes the distribution) plus a recent window. **Adaptive KV eviction** in some 2025-2026 serving stacks combines heavy-hitter scoring with chunked drops to reduce per-step overhead. The practical choice is a quality vs bookkeeping trade. Use sliding-window when you need cheap and predictable; use heavy-hitter when long-range references matter and quality at extreme context is the binding constraint.

Quick reference

TL;DR

Two production policies: sliding-window (keep last W, free bookkeeping) and heavy-hitter (keep tokens with highest accumulated attention weight, per-token score tracking).

Memory aid

Easy to grasp

Picture a chef's prep station that can only hold 20 ingredients at a time. As new ingredients come in, old ones have to go. The simple rule is 'whatever has been sitting longest, throw it out',that is sliding-window. The smarter rule is 'whatever the recipes keep reaching for, keep it; whatever has been ignored, throw it out',that is heavy-hitter. The smarter rule needs the chef to keep a tally of how often each ingredient gets used, but it preserves the important stuff better. Production kitchens often blend the two: always keep a few staples like salt at the start, and otherwise rotate the recent and the popular.

Key concepts

KV cache Sliding window attention H2O eviction Attention sink StreamingLLM

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Name sliding-window eviction (keep last W, free bookkeeping) and heavy-hitter eviction (keep top-k by attention score, per-token bookkeeping). Cite StreamingLLM as the hybrid and explain attention sinks. Close with the production decision rubric: bounded context vs streaming vs extreme long context.

Real-world examples

Real products, models, and research that use this idea.

Mistral 7B: sliding-window attention with W = 4096 is the architectural form of sliding-window eviction.
H2O (NeurIPS 2023): the canonical heavy-hitter eviction paper; showed near-lossless quality at 20% cache size on long-context QA benchmarks.
StreamingLLM (Xiao et al., 2024): attention-sink + recent-window hybrid, deployed widely in 2024-2026 long-context serving stacks.
DeepSeek V4 multi-head latent attention: not eviction but related, compresses KV per token to shrink the budget pressure that eviction tries to address.
vLLM and SGLang in 2026: support sliding-window and StreamingLLM-style hybrids as configurable serving policies.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does dropping the first few tokens collapse the attention distribution in long generations?

QHow does heavy-hitter eviction interact with grouped-query attention (GQA) where K and V are shared across query heads?

QWhen does KV quantization beat eviction at the same memory budget?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating KV cache eviction as a pure capacity problem with one right answer. Quality varies by eviction policy because not all cached tokens are equally useful for future generation.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.