Name two recovery moves a serving stack can pull when KV memory runs short mid-request.

Short answer·Medium·4.0 · 0·~3 min

Attempt it

A long-context serving job is about to exceed its KV cache budget mid-generation. The scheduler can either kill the request or degrade quality gracefully. Name two attention-side strategies the runtime can apply, and explain the tradeoff each makes.

Free · 2 AI evals / day

When the **KV cache** runs out of memory mid-generation, the scheduler has two main attention-side recovery moves: drop entries or shrink each entry. **Eviction**: drop cached tokens that probably will not be attended to again. The simplest version is a sliding window, keep only the most recent W tokens and discard older ones. A smarter version called H2O ranks tokens by how much total attention weight they have received and keeps the top-k plus the most recent. StreamingLLM is a popular hybrid, it keeps a few initial sink tokens permanently plus a sliding window. The tradeoff: any information past the kept set is gone, so long-range references silently fail. Quality degrades smoothly rather than crashing. **In-place quantization**: re-encode the FP16 KV cache to INT8 or INT4 on the fly, halving or quartering the memory footprint without dropping any tokens. The tradeoff is precision, rounding errors compound across attention layers and outlier channels can clip badly without per-channel scales. Quality drop is broad-spectrum, not specifically long-range. Both beat killing the request. Real systems combine them.

The two recovery moves are **eviction** and **in-place quantization**. They attack orthogonal axes: number of cache entries and bytes per entry. **Eviction** reduces the entry count. Three common policies: - **Sliding window**: keep the last W tokens, discard the rest. Simplest, cheapest, works surprisingly well for chat workloads. - **H2O (Heavy Hitters)**: keep tokens by accumulated attention weight, plus the most recent. Smarter than sliding window when the relevant context is concentrated in a few distant tokens. - **StreamingLLM**: keep the first 4 tokens permanently (the attention sinks) plus a sliding window. Solves the 'first-token degradation' that pure sliding window suffers from. The tradeoff: anything past the kept set is irrecoverably gone. Long-range references fail silently, the model just answers as if the context never existed. Quality degrades smoothly without a hard crash. **In-place quantization** reduces bytes per entry. Convert the FP16 KV cache to INT8 or INT4 with per-channel or per-token scaling factors. INT8 halves memory, INT4 quarters it. No tokens are dropped, all positions remain attendable. The tradeoff is precision. Each attention layer reads and writes the cache, so quantization noise accumulates layer by layer. Outlier channels (a known property of attention activations) clip badly without careful per-channel scales. KIVI and similar schemes mitigate this with mixed-precision approaches. Real serving stacks (vLLM, TensorRT-LLM, SGLang) combine both, often quantizing the cache early and applying eviction policies under sustained pressure. Both options beat the alternative of OOM-killing the request and forcing a client retry.

Quick reference

TL;DR

Eviction (drop tokens, sliding window or H2O) and in-place quantization (FP16 to INT8 or INT4). Both degrade quality gracefully instead of OOM-killing the request.

Memory aid

Easy to grasp

Picture a bookshelf running out of room while you are still adding books. Two ways to make space without throwing the whole shelf out. First, toss old books you have not opened in a while, you lose some content but you keep reading. Second, photocopy each book into smaller print so each one takes half the space, you keep every book but the print is harder to read. A serving system's short-term memory works the same way: drop notes you probably will not need (call it eviction) or store each note using fewer digits per number (call it compression). Real systems usually do both before giving up and dropping the whole job.

Key concepts

KV cache KV cache eviction KV cache quantization StreamingLLM

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6-8 min: KV memory equation + eviction policies (sliding, H2O, StreamingLLM) + quantization policies (INT8, INT4, FP8) + composition in real serving stacks + the role of PagedAttention.

Side-by-side

Strategy	Axis attacked	Quality tradeoff	Implementation cost
Sliding window	Entry count (T)	Drops oldest context	Trivial
H2O	Entry count (T)	Drops low-attention tokens	Per-token running sum
StreamingLLM	Entry count (T)	Window + sink tokens	Trivial extension of sliding
INT8 KV	Bytes per entry (b)	Compounds across layers	Per-channel scales
INT4 KV	Bytes per entry (b)	Stronger precision loss	Per-group scales (KIVI)

Real-world examples

Real products, models, and research that use this idea.

vLLM with PagedAttention plus INT8 KV cache support combines block-level memory layout with quantization.
TensorRT-LLM ships INT8 and FP8 KV cache modes targeted at H100 and newer GPUs.
StreamingLLM (the 'attention sinks' paper, MIT 2023) and H2O (2023) are the canonical eviction references cited in serving stack docs.
Llama 4 Maverick, DeepSeek V4, and Qwen 3.5 long-context configurations rely on GQA plus FP8 or INT8 KV quantization for fleet-scale serving.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does pure sliding window degrade quality even when window is generous?

QWhat is PagedAttention and how does it relate to KV cache management?

QCan you combine eviction with continuous batching?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Forgetting that long-context serving has two orthogonal shrink axes (entry count and bytes per entry) and naming only one. A senior answer mentions both.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.