Name two recovery moves a serving stack can pull when KV memory runs short mid request.

Short answer·Medium·4.0 · 0·~3 min·Asked atCognizant Dify Krutrim·Relevant atAi4bharat Anthropic Cerebras Deepseek

Attempt it

A long context serving job is about to exceed its KV cache budget mid generation. The scheduler can either kill the request or degrade quality gracefully. Name two attention side strategies the runtime can apply, and explain the tradeoff each makes.

Free · 2 AI evals / day

When the **KV cache** runs out of memory mid-generation, the scheduler has two main attention-side recovery moves: drop entries or shrink each entry. **Eviction**: drop cached tokens that probably will not be attended to again. The simplest version is a sliding window, keep only the most recent W tokens and discard older ones. A smarter version called H2O ranks tokens by how much total attention weight they have received and keeps the top-k plus the most recent. StreamingLLM is a popular hybrid, it keeps a few initial sink tokens permanently plus a sliding window. The tradeoff: any information past the kept set is gone, so long-range references silently fail. Quality degrades smoothly rather than crashing. **In-place quantization**: re-encode the FP16 KV cache to INT8 or INT4 on the fly, halving or quartering the memory footprint without dropping any tokens. The tradeoff is precision, rounding errors compound across attention layers and outlier channels can clip badly without per-channel scales. Quality drop is broad-spectrum, not specifically long-range. Both beat killing the request. Real systems combine them.

The two recovery moves are **eviction** and **in-place quantization**. They attack orthogonal axes: number of cache entries and bytes per entry. **Eviction** reduces the entry count. Three common policies: - **Sliding window**: keep the last W tokens, discard the rest. Simplest, cheapest, works surprisingly well for chat workloads. - **H2O (Heavy Hitters)**: keep tokens by accumulated attention weight, plus the most recent. Smarter than sliding window when the relevant context is concentrated in a few distant tokens. - **StreamingLLM**: keep the first 4 tokens permanently (the attention sinks) plus a sliding window. Solves the 'first-token degradation' that pure sliding window suffers from. The tradeoff: anything past the kept set is irrecoverably gone. Long-range references fail silently, the model just answers as if the context never existed. Quality degrades smoothly without a hard crash. **In-place quantization** reduces bytes per entry. Convert the FP16 KV cache to INT8 or INT4 with per-channel or per-token scaling factors. INT8 halves memory, INT4 quarters it. No tokens are dropped, all positions remain attendable. The tradeoff is precision. Each attention layer reads and writes the cache, so quantization noise accumulates layer by layer. Outlier channels (a known property of attention activations) clip badly without careful per-channel scales. KIVI and similar schemes mitigate this with mixed-precision approaches. Real serving stacks (vLLM, TensorRT-LLM, SGLang) combine both, often quantizing the cache early and applying eviction policies under sustained pressure. Both options beat the alternative of OOM-killing the request and forcing a client retry.

TL;DR

Eviction (drop tokens, sliding window or H2O) and in-place quantization (FP16 to INT8 or INT4). Both degrade quality gracefully instead of OOM-killing the request.

Memory aid

Easy to grasp

Picture a bookshelf running out of room while you are still adding books. Two ways to make space without throwing the whole shelf out. First, toss old books you have not opened in a while, you lose some content but you keep reading. Second, photocopy each book into smaller print so each one takes half the space, you keep every book but the print is harder to read. A serving system's short-term memory works the same way: drop notes you probably will not need (call it eviction) or store each note using fewer digits per number (call it compression). Real systems usually do both before giving up and dropping the whole job.

Key concepts

KV cache KV cache eviction KV cache quantization StreamingLLM

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

KV cache memory is the single biggest serving-time constraint for long-context LLM inference. When the cache budget runs out mid-generation, the scheduler has a choice: kill the request and force a retry, or degrade gracefully. Two attention-side strategies make graceful degradation possible.

This deep dive walks through the memory equation, the two independent shrink axes, the canonical eviction and quantization policies, and how production serving stacks combine them in practice.

Mental model: KV memory is the product of entry count and bytes per entry. Eviction attacks the first; quantization attacks the second. Real stacks do both before they give up.

The KV memory equation

What the cache stores

For each generated token, every transformer layer caches its K and V tensors so future tokens can attend to them without recomputing. With Multi-Query or Grouped-Query Attention, fewer KV heads are stored than Q heads, but the per-token cost is otherwise straightforward.

The exact formula

\text{bytes} = 2 \cdot L \cdot H_{kv} \cdot d_h \cdot T \cdot b

Where:

2 = one tensor for K, one for V.
L = number of layers.
H_kv = number of KV heads (reduced by MQA or GQA).
d_h = head dimension (usually 128).
T = sequence length (the term that scales linearly with context).
b = bytes per element (2 for FP16, 1 for INT8, 0.5 for INT4).

Concrete example

For Llama-3-70B with 80 layers, 8 KV heads (GQA), d_h = 128, T = 8192, FP16:

2 * 80 * 8 * 128 * 8192 * 2 = 2.68 GB per request

Multiply by batch size and concurrent sequences and the cache dominates GPU memory before model weights do.

The two attackable axes

From the formula, only T (entry count) and b (bytes per entry) are runtime-modifiable. L, H_kv, and d_h are baked into model weights. Eviction shrinks T. Quantization shrinks b.

Eviction: shrink the entry count

Quantization: shrink the bytes per entry

Composing the strategies in production

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Strategy	Axis attacked	Quality tradeoff	Implementation cost
Sliding window	Entry count (T)	Drops oldest context	Trivial
H2O	Entry count (T)	Drops low-attention tokens	Per-token running sum
StreamingLLM	Entry count (T)	Window + sink tokens	Trivial extension of sliding
INT8 KV	Bytes per entry (b)	Compounds across layers	Per-channel scales
INT4 KV	Bytes per entry (b)	Stronger precision loss	Per-group scales (KIVI)

Real products, models, and research that use this idea.

vLLM with PagedAttention plus INT8 KV cache support combines block-level memory layout with quantization.
TensorRT-LLM ships INT8 and FP8 KV cache modes targeted at H100 and newer GPUs.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does pure sliding window degrade quality even when window is generous?

Models trained with full attention learn to route excess softmax probability to the first few tokens (attention sinks). Drop those sinks via sliding and the first kept token inherits the sink role, distorting its representation. StreamingLLM fixes this by keeping the sinks permanently.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Forgetting that long-context serving has two orthogonal shrink axes (entry count and bytes per entry) and naming only one. A senior answer mentions both.

60 second bullets to scan on the way to the call.

Two independent axes of KV memory: entry count and bytes per entry
Sliding window, H2O, and StreamingLLM eviction policies

Primary sources. Browse if you want the original framing.