Pair KV-cache eviction and KV quantization with the situation each one is the right answer for

Match pairs·Medium·4.0 · 0·~2 min·Asked atFiddler Ai NVIDIA Pinterest

Attempt it

Drag each answer to line up with its matching prompt

Eviction (StreamingLLM style sink plus sliding window)

Eviction reduces the NUMBER of cached tokens; quantization reduces the BYTES per cached token

KV quantization (FP8 or INT8 KV)

Eviction loses information about evicted tokens entirely; quantization degrades all tokens slightly via precision loss

What you save

Sessions of unbounded length where you accept losing distant context to keep the cache from growing without bound

Composability

Bounded length context where you want every step to read fewer bytes and accept a small quality cost from lower precision

Quality failure mode

The two policies stack: FP8 quantized KV with a sink plus sliding eviction window is a common long session production setup

The KV cache holds the keys and values for every token seen so far. Its size is a product: number of tokens times bytes per token (per layer, per head, etc.). Two techniques shrink it along different axes. **Eviction** throws away cached entries for old tokens. The simplest scheme is a sliding window: keep only the last N tokens. StreamingLLM adds a twist: also keep the first few tokens (the 'attention sinks') because models learn to dump attention mass on them, and dropping them destabilizes generation. Eviction is the right answer when the conversation runs forever and the cache would otherwise grow without bound. The price is that you lose information about evicted tokens entirely. **KV quantization** keeps every cached entry but stores it in a lower-precision format. The standard recipe is FP8 or INT8 instead of FP16: half the bytes per cached value. The cache still grows linearly with context length, just at a smaller per-token cost. The price is a small quality drop because the cached values lose some precision, but every token is still represented. The two are not alternatives; they stack. A long-session production deployment often runs FP8 KV quantization plus a StreamingLLM-style sliding window with sinks: half the bytes per token AND a bounded token count. Pick eviction when the limit is total tokens (unbounded session length). Pick quantization when the limit is bytes (per-step bandwidth or HBM capacity at a bounded context length). Use both when both limits bite.

The KV cache size is the product of two factors: how many tokens it stores and how many bytes each cached token uses. Eviction shrinks the first factor; quantization shrinks the second. The choice between them is usually a workload-shape question. **KV eviction** drops cached entries for some tokens entirely. Variants include a pure sliding window (keep the last N tokens), StreamingLLM (last N plus the first few "attention sinks" because models develop strong dependence on early tokens regardless of content), H2O / Heavy-Hitter Oracle (keep the most attended-to tokens dynamically), and a growing family of learned eviction policies. Eviction is the right tool when sessions can grow unboundedly long (multi-hour chat, agent loops, continuous streaming) because without it the cache eventually exceeds HBM. The quality cost is informational: any question about an evicted token's content cannot be answered from the cache. **KV quantization** stores cached keys and values at lower precision. Common targets are FP8 (1 byte per value, the production default in 2026), INT8 (1 byte, integer math), and INT4 (0.5 bytes, more aggressive with measurable quality cost). The cache still grows linearly with token count, just at half or a quarter of the per-token bytes. Quantization is the right tool when the context length is bounded but the per-step bandwidth (or total HBM footprint at long context) is the binding constraint. The quality cost is precision-driven: all cached values are slightly noisy. The two compose cleanly. A long-session production deployment in 2026 typically runs FP8-quantized KV with a StreamingLLM-style window: the bytes per token are halved AND the token count is bounded by the window. Some stacks layer H2O-style importance retention on top to keep the right tokens within the window. The failure modes differ in flavor. Eviction has a hard cliff: evicted tokens are gone, and any reasoning that depended on them produces wrong answers without warning.

TL;DR

KV eviction reduces the NUMBER of cached tokens (drop old ones); KV quantization reduces the BYTES per cached token (lower precision).

Memory aid

Easy to grasp

Imagine a filing cabinet that fills up as a conversation grows. Two ways to make it fit. The first approach throws out old folders to make room, but those folders are gone for good, so anything you needed from them is forgotten. The second approach keeps every folder but writes the contents in a smaller, slightly smudgier font so each folder is thinner. The cabinet is still full of every file, but each file is less detailed. The throwing-out trick is what you want when the cabinet would otherwise overflow forever (a never-ending chat); the smaller-font trick is what you want when the cabinet fits but is still too slow to flip through. You can combine both: throw out old folders AND write the rest in the smaller font.

Key concepts

KV cache KV quantization KV eviction StreamingLLM

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The KV cache is the dominant per-request memory cost in long-context decode, and there are two distinct ways to shrink it. Mixing them up is one of the most common interview tells for someone who has not deployed long-context serving.

The cache size is a product. Bytes equals number of cached tokens times bytes per cached token. Each factor has its own optimization family. Eviction (sliding window, StreamingLLM, H2O) reduces the first factor. KV quantization (FP8, INT8, INT4) reduces the second. They are not alternatives; they target different dimensions and compose multiplicatively.

Which one you pull off the shelf first depends on the binding constraint. If sessions can grow without bound, no per-token saving will save you eventually; you have to bound the count. If sessions are bounded but long, the count is fixed and the binding cost is per-step bandwidth, which scales with bytes per token. The right answer for production is usually both.

This deep dive walks the cache-size factoring, the eviction policy ladder, the KV quantization ladder, and the composability that makes long-session frontier serving economically viable in 2026.

The KV cache factoring: tokens times bytes per token

The KV cache memory cost for a single request follows a clean formula:

\text{bytes} = 2 \cdot L \cdot H_{kv} \cdot d_h \cdot T \cdot b

where L is the layer count, H_kv is the KV-head count (varies with MHA / GQA / MQA), d_h is the per-head dimension, T is the current token count, and b is bytes per element. The factor of 2 covers keys and values.

Group the terms. The architectural factors (L, H_kv, d_h) are fixed at model design time. The two factors you can change at serving time are T (the token count) and b (bytes per element). That is exactly where eviction and quantization act.

Eviction reduces T. Instead of growing the cache linearly with session length, eviction enforces an upper bound on T regardless of how long the session runs. KV quantization reduces b. Instead of 2 bytes per element (FP16 / BF16), you store 1 byte (FP8 / INT8) or 0.5 bytes (INT4). The cache still grows linearly with token count, just at a smaller per-token cost.

Because T and b are independent factors of the same product, the two optimizations multiply. FP8 KV (b cut in half) with a 4k sliding window (T capped at 4096) gives total cache use of approximately (4096) * (1 byte / element) * (architectural factors), versus the original (session_length) * (2 bytes / element) * (architectural factors). For a long session this is orders of magnitude smaller and remains bounded regardless of how long the session continues.

Eviction policies: sliding window, StreamingLLM, H2O

KV quantization recipes: FP8, INT8, INT4

The composability that makes long-session serving work

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	KV eviction	KV quantization
Dimension reduced	Number of cached tokens	Bytes per cached token
Right workload	Unbounded session length	Bounded but long context
Quality failure mode	Hard cliff: evicted tokens are gone	Soft slope: all tokens slightly noisy
Typical 2026 recipe	StreamingLLM sink plus sliding	FP8 with per-channel calibration
Saving	Bounded cache size regardless of session length	2x at FP8, 4x at INT4
Stacks with the other?	Yes	Yes

Real products, models, and research that use this idea.

vLLM ships both FP8 KV quantization and configurable eviction policies (sliding window, StreamingLLM-style sinks) as orthogonal flags.
SGLang's long-session serving for chat agents combines FP8 KV with attention-sink retention to handle multi-hour conversations without OOM.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy do attention sinks form on the first few tokens, and what happens if you evict them?

Models learn to use early tokens as a dump for excess attention mass because softmax must sum to 1 even when no later tokens deserve attention. If you evict them, the attention distribution destabilizes (the model has nowhere to dump the mass) and generation quality collapses, often into repetitive or incoherent output. StreamingLLM keeps the first K tokens precisely to preserve this attention sink.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating the two as competing techniques. They attack different dimensions of the cache (token count versus bytes per token) and stack cleanly: a long-session production setup often runs FP8-quantized KV with a StreamingLLM-style sink plus sliding eviction policy.

60 second bullets to scan on the way to the call.

Why the KV cache size factors into (tokens) * (bytes per token)
Which dimension eviction trims and which quantization trims

Primary sources. Browse if you want the original framing.