Pair KV-cache eviction and KV-quantization with the situation each one is the right answer for

Match pairs·Medium·4.0 · 0·~2 min·Relevant atNVIDIA

Attempt it

Eviction (StreamingLLM-style sink plus sliding window)

KV quantization (FP8 or INT8 KV)

What you save

Composability

Quality failure mode

The KV cache holds the keys and values for every token seen so far. Its size is a product: number of tokens times bytes per token (per layer, per head, etc.). Two techniques shrink it along different axes. **Eviction** throws away cached entries for old tokens. The simplest scheme is a sliding window: keep only the last N tokens. StreamingLLM adds a twist: also keep the first few tokens (the 'attention sinks') because models learn to dump attention mass on them, and dropping them destabilizes generation. Eviction is the right answer when the conversation runs forever and the cache would otherwise grow without bound. The price is that you lose information about evicted tokens entirely. **KV quantization** keeps every cached entry but stores it in a lower-precision format. The standard recipe is FP8 or INT8 instead of FP16: half the bytes per cached value. The cache still grows linearly with context length, just at a smaller per-token cost. The price is a small quality drop because the cached values lose some precision, but every token is still represented. The two are not alternatives; they stack. A long-session production deployment often runs FP8 KV quantization plus a StreamingLLM-style sliding window with sinks: half the bytes per token AND a bounded token count. Pick eviction when the limit is total tokens (unbounded session length). Pick quantization when the limit is bytes (per-step bandwidth or HBM capacity at a bounded context length). Use both when both limits bite.

The KV cache size is the product of two factors: how many tokens it stores and how many bytes each cached token uses. Eviction shrinks the first factor; quantization shrinks the second. The choice between them is usually a workload-shape question. **KV eviction** drops cached entries for some tokens entirely. Variants include a pure sliding window (keep the last N tokens), StreamingLLM (last N plus the first few "attention sinks" because models develop strong dependence on early tokens regardless of content), H2O / Heavy-Hitter Oracle (keep the most attended-to tokens dynamically), and a growing family of learned eviction policies. Eviction is the right tool when sessions can grow unboundedly long (multi-hour chat, agent loops, continuous streaming) because without it the cache eventually exceeds HBM. The quality cost is informational: any question about an evicted token's content cannot be answered from the cache. **KV quantization** stores cached keys and values at lower precision. Common targets are FP8 (1 byte per value, the production default in 2026), INT8 (1 byte, integer math), and INT4 (0.5 bytes, more aggressive with measurable quality cost). The cache still grows linearly with token count, just at half or a quarter of the per-token bytes. Quantization is the right tool when the context length is bounded but the per-step bandwidth (or total HBM footprint at long context) is the binding constraint. The quality cost is precision-driven: all cached values are slightly noisy. The two compose cleanly. A long-session production deployment in 2026 typically runs FP8-quantized KV with a StreamingLLM-style window: the bytes per token are halved AND the token count is bounded by the window. Some stacks layer H2O-style importance retention on top to keep the right tokens within the window. The failure modes differ in flavor. Eviction has a hard cliff: evicted tokens are gone, and any reasoning that depended on them produces wrong answers without warning.

Quick reference

TL;DR

KV eviction reduces the NUMBER of cached tokens (drop old ones); KV quantization reduces the BYTES per cached token (lower precision).

Memory aid

Easy to grasp

Imagine a filing cabinet that fills up as a conversation grows. Two ways to make it fit. The first approach throws out old folders to make room, but those folders are gone for good, so anything you needed from them is forgotten. The second approach keeps every folder but writes the contents in a smaller, slightly smudgier font so each folder is thinner. The cabinet is still full of every file, but each file is less detailed. The throwing-out trick is what you want when the cabinet would otherwise overflow forever (a never-ending chat); the smaller-font trick is what you want when the cabinet fits but is still too slow to flip through. You can combine both: throw out old folders AND write the rest in the smaller font.

Key concepts

KV cache KV quantization KV eviction StreamingLLM

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: factor the cache as (tokens) * (bytes per token), assign eviction and quantization to each dimension, identify unbounded session versus bounded long context as the selector, and explicitly state that they stack with a StreamingLLM-plus-FP8 example.

Side-by-side

Property	KV eviction	KV quantization
Dimension reduced	Number of cached tokens	Bytes per cached token
Right workload	Unbounded session length	Bounded but long context
Quality failure mode	Hard cliff: evicted tokens are gone	Soft slope: all tokens slightly noisy
Typical 2026 recipe	StreamingLLM sink plus sliding	FP8 with per-channel calibration
Saving	Bounded cache size regardless of session length	2x at FP8, 4x at INT4
Stacks with the other?	Yes	Yes

Real-world examples

Real products, models, and research that use this idea.

vLLM ships both FP8 KV quantization and configurable eviction policies (sliding window, StreamingLLM-style sinks) as orthogonal flags.
SGLang's long-session serving for chat agents combines FP8 KV with attention-sink retention to handle multi-hour conversations without OOM.
Claude Opus 4.7's deployed long-context handling uses KV quantization plus aggressive prefix caching to make 200k-token contexts economically tractable.
TensorRT-LLM exposes per-layer KV quantization (FP8, INT8) as a build-time option, with eviction handled at the runtime scheduler level.
H2O (Heavy-Hitter Oracle) and SnapKV introduced importance-based eviction policies that retain the most-attended tokens within a fixed budget, used in research-grade long-context stacks.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy do attention sinks form on the first few tokens, and what happens if you evict them?

QHow does importance-based eviction (H2O, SnapKV) differ from sliding window?

QWhen does INT4 KV quantization stop being worth the saving?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating the two as competing techniques. They attack different dimensions of the cache (token count versus bytes per token) and stack cleanly: a long-session production setup often runs FP8-quantized KV with a StreamingLLM-style sink plus sliding eviction policy.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.