Memory creeps up for hours then OOMs at 3am: diagnose the KV-cache leak pattern

Short answer·Medium·4.0 · 0·~3 min·Asked atAirbnb Capgemini Writer·Relevant atNVIDIA

Attempt it

A vLLM style serving pod shows healthy GPU memory at startup. Over six to twelve hours, KV-cache utilization climbs monotonically, the scheduler admits fewer concurrent requests, latency rises, and eventually the pod OOMs. Request volume is steady throughout. Diagnose the pattern: what is leaking, what is the upstream trigger, and what three mitigations would you put in place?

Free · 2 AI evals / day

**The leak is in KV cache pages tied to abandoned client streams.** Here is what happens in slow motion. Each inference request reserves a chunk of GPU memory for its KV cache. The reservation is held until the request completes normally (EOS token) or is explicitly cancelled. The serving framework, like vLLM or TensorRT-LLM, returns the pages to a free pool at one of those two events. When a user closes a browser tab, kills the network on their phone, or a load balancer times out a long SSE connection, no cancel signal reaches the inference engine. The request is, from the engine's perspective, still alive and still streaming. Over six to twelve hours of normal traffic, a small percentage of these orphans accumulate. The free pool shrinks. The scheduler can admit fewer concurrent requests, so latency climbs even though traffic is steady. Eventually the next admission cannot find enough free pages and the pod OOMs. The fix is three independent defenses: an idle-sequence reaper that frees sequences with no token progress for N seconds, a hard max session age cap, and cancel propagation from the gateway down to the inference API.

**Diagnosis.** The leaking entity is the per-sequence KV cache. Pages are held until the scheduler sees EOS or a cancel call. **The upstream trigger is broken client streams** where the gateway, load balancer, or browser tear-down does not propagate to the inference server. The symptom signature is distinctive. Memory grows monotonically at steady traffic. Free-page count drops over hours. Admitted-concurrency falls in lockstep, so throughput drops and queue depth rises even though incoming RPS is flat. The pod OOMs at the moment the next admission cannot allocate. Reproducible by running a load test that opens streaming connections and then drops them at random. Three mitigations, in priority order: 1. **Cancel propagation**. Wire the HTTP or SSE disconnect event from the gateway and load balancer through to the inference server's cancel endpoint. This is the root-cause fix; the others are safety nets. 2. **Idle-sequence reaper**. A watchdog that scans active sequences for ones with no token-generation progress in 30-60 seconds and force-frees them. Catches the failure case where the cancel propagation also fails. 3. **Hard max session age**. A flat cap (e.g., 5 minutes) past which any sequence is killed regardless of progress. Catches truly stuck sequences, runaway agent loops, and malicious clients holding TCP alive. The bug usually lives in the gateway to inference gap. The gateway knows the client is gone within seconds. The inference engine never finds out unless someone explicitly tells it.

TL;DR

Orphaned sequences from abandoned client streams pin KV cache pages forever. Fix with idle reaper, max session age, and end to end cancel propagation from gateway to inference server.

Memory aid

Easy to grasp

Imagine a coat-check at a busy restaurant. Each guest takes a numbered hanger when they arrive and returns it on the way out. If guests slip out a back door without claiming their coat, the hangers fill up. After enough nights, the rack has no free hangers and new guests get turned away even though the dining room is empty. The KV cache works the same way. Each chat request rents some pages of GPU memory to store its conversation state. The pages come back when the request ends cleanly. But if the user closes the browser tab, the inference engine never gets the end signal, the pages stay rented forever, and the rack slowly fills up. After hours of small leaks, there is no free space left and the pod crashes.

Key concepts

KV cache Model serving Memory leak

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

This is a classic production failure pattern in LLM serving: memory drifts up over hours at steady traffic, then the pod OOMs in the middle of the night. The symptom is recognizable, the cause is consistent, and the fix is a three-layer defense rather than a single patch.

The diagnosis hinges on understanding that the KV cache is owned by individual sequences and released only on explicit events. When those events do not fire, pages pile up. The trigger is almost always client-side disconnects that do not propagate down to the inference engine.

This walkthrough covers the leak mechanism end to end: what the KV cache pool looks like in a paged-attention serving stack, why orphaned sequences accumulate, how the symptom manifests in metrics, the three-layer mitigation strategy, and the adversarial variants that one of the layers exists specifically to catch.

By the end you should be able to recognize the pattern within minutes of looking at the free page count metric, locate the bug in the gateway to inference plumbing layer, and prescribe defenses that hold up under network failures, runaway agents, and malicious clients alike.

How paged KV cache allocation actually works

Modern inference engines (vLLM, TensorRT-LLM, SGLang) manage KV cache as a paged pool. The HBM left over after model weights and activations is carved into fixed-size pages, each holding the K and V vectors for a small block of tokens (typically 16 tokens per page).

When a request arrives, the scheduler reserves enough pages for the prefill plus a buffer for early decode. As decode proceeds, more pages are claimed from the free pool. When the request completes, its pages return to the pool.

Two things complete a request: the model emits an EOS token (or the configured max output tokens cap is hit), or the client explicitly cancels via API. Both result in the scheduler calling the per-sequence cleanup path that returns pages to the free list.

If neither event fires, the sequence sits in the active set indefinitely. The scheduler keeps trying to generate the next token for it, the model keeps producing logits, and the pages keep being held.

The leak mechanism

The symptom signature

Three-layer mitigation strategy

Adversarial variants and why max-age is non-negotiable

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's Claude Opus 4.7 serving fleet wires Envoy SSE disconnect events directly to the inference cancel API to prevent this exact failure.
vLLM 0.6+ ships a built-in finished-flag watchdog that catches sequences with no progress for a configurable interval.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is cancel propagation surprisingly hard to get right across a modern service mesh?

TCP RST does not survive multiple proxy hops cleanly. Each layer (CDN, ingress, service mesh sidecar, gateway) terminates the connection and re-proxies, so the disconnect event is local to each hop. Application-layer cancel must be carried as an explicit signal, often through a shared request-id channel or out of band cancel API. Plus version-skew between gateway and inference can drop the cancel.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Blaming the model or scheduler. The leak almost always lives in the gateway to inference cancel path, where client disconnects are silently dropped.

60 second bullets to scan on the way to the call.

Name the leaking entity and what event normally releases it
Describe how a client disconnect can fail to propagate to the inference server