Memory creeps up for hours then OOMs at 3am: diagnose the KV-cache leak pattern

Short answer·Medium·4.0 · 0·~3 min·Asked atAirbnb Writer·Relevant atNVIDIA

Attempt it

A vLLM style serving pod shows healthy GPU memory at startup. Over six to twelve hours, KV-cache utilization climbs monotonically, the scheduler admits fewer concurrent requests, latency rises, and eventually the pod OOMs. Request volume is steady throughout. Diagnose the pattern: what is leaking, what is the upstream trigger, and what three mitigations would you put in place?

Free · 2 AI evals / day

**The leak is in KV cache pages held by abandoned client streams.** Each inference request reserves GPU memory for its KV cache. The serving framework (vLLM, TensorRT-LLM) returns those pages to a free pool when the request ends: an EOS token, the max-token cap or an explicit cancel. When a user closes a browser tab, loses the network on a phone, or a load balancer times out a long SSE connection, no cancel reaches the inference engine. To the engine, the request is still alive. Most of these orphans still end at the max-token cap, but any that stall never release their pages. Over six to twelve hours of steady traffic, these stalled orphans pile up. The free pool shrinks. The scheduler admits fewer concurrent requests, so latency climbs even though traffic is steady. Eventually the next sequence cannot get enough free pages and the pod fails. The fix is three independent defenses: cancel propagation from the gateway down to the inference API, an idle-sequence reaper that frees sequences with no token progress for N seconds, and a hard cap on session age.

**Diagnosis.** The leaking entity is the per-sequence KV cache. Pages are held until the scheduler sees EOS, the max-token cap or a cancel call. **The upstream trigger is broken client streams**: a browser close, a load-balancer timeout or a proxy drop that never reaches the inference server as a cancel. Most orphans still end at the max-token cap, but any that stall hold their pages indefinitely. Free-page count drops over hours. Admitted concurrency falls with it, so throughput drops and queue depth rises even though incoming RPS is flat. The pod fails when the next sequence cannot get a page. A load test that opens streaming connections and then drops them at random reproduces the pattern. Three mitigations, in priority order: 1. **Cancel propagation**. Wire the HTTP or SSE disconnect event from the gateway and load balancer through to the inference server's cancel path. This is the root-cause fix; the others are safety nets. 2. **Idle-sequence reaper**. A watchdog frees active sequences with no token-generation progress in 30-60 seconds. It catches stalled orphans when cancel propagation also fails. 3. **Hard max session age**. A flat cap (for example 5 minutes) past which any sequence is killed regardless of progress. It catches stuck sequences, runaway generations and clients that hold TCP open. The bug usually lives in the gateway-to-inference gap. The gateway knows within seconds that the client is gone. The inference engine never finds out unless something tells it.

TL;DR

Orphaned sequences from abandoned client streams hold KV cache pages long after the client is gone. Fix with idle reaper, max session age, and end-to-end cancel propagation from gateway to inference server.

Memory aid

Easy to grasp

Imagine a coat-check at a busy restaurant. Each guest takes a numbered hanger when they arrive and returns it on the way out. If guests slip out a back door without claiming their coat, the hangers fill up. After enough nights, the rack has no free hangers and new guests get turned away even though the dining room is empty. The KV cache works the same way. Each chat request rents some pages of GPU memory to store its conversation state. The pages come back when the request ends cleanly. But if the user closes the browser tab, the inference engine never gets the end signal. The pages can stay rented for a long time, and the rack slowly fills up. After hours of small leaks, there is no free space left and the pod crashes.

Key concepts

KV cache Model serving Memory leak

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

This is a classic production failure pattern in LLM serving: memory drifts up over hours at steady traffic, then the pod OOMs in the middle of the night. The symptom is recognizable, the cause is consistent, and the fix is a three-layer defense rather than a single patch.

The diagnosis depends on one fact: individual sequences own the KV cache, and it is released only on explicit events. When those events do not fire, pages pile up. The trigger is almost always client-side disconnects that do not propagate down to the inference engine.

The free-page count shows the pattern early. The bug usually sits in the path that carries a cancel from the gateway to the inference engine.

How paged KV cache allocation works

Inference engines such as vLLM, TensorRT-LLM and SGLang manage KV cache as a paged pool. The HBM left over after model weights and activations is split into fixed-size pages. Each page holds the K and V vectors for a small block of tokens (typically 16 tokens per page).

When a request arrives, the scheduler reserves enough pages for the prefill plus a buffer for early decode. As decode proceeds, more pages are claimed from the free pool. When the request completes, its pages return to the pool.

Two things complete a request: the model emits an EOS token (or the configured max output tokens cap is hit), or the client explicitly cancels via API. In both cases, the scheduler calls the per-sequence cleanup path that returns pages to the free list.

If no cancel arrives, the scheduler keeps generating tokens for the sequence and its pages stay held until EOS or the output cap. If the sequence stalls, or a bug skips the cleanup path, it stays in the active set and its pages are never returned.

The leak mechanism

The symptom signature

Three-layer mitigation strategy

Adversarial variants that only the max-age cap catches

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM's OpenAI-compatible server wraps its route handlers in a with_cancellation decorator. When the server sees an HTTP disconnect, it cancels the handler and the request is aborted. Orphans appear when a proxy in front keeps the upstream connection open after the client has left.
nginx closes its connection to the upstream server when the client closes the connection. The exception is when proxy_ignore_client_abort is turned on, which keeps the upstream request running for a client that is gone.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is cancel propagation hard to get right across a service mesh?

TCP RST does not survive multiple proxy hops cleanly. Each layer (CDN, ingress, service mesh sidecar, gateway) terminates the connection and re-proxies, so the disconnect event is local to each hop. Application-layer cancel must be carried as an explicit signal, often through a shared request-id channel or an out-of-band cancel API. Version skew between gateway and inference server can also drop the cancel.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Blaming the model or scheduler. The leak almost always lives in the gateway-to-inference cancel path, where client disconnects are silently dropped.

60 second bullets to scan on the way to the call.

Name the leaking entity and what event normally releases it
Describe how a client disconnect can fail to propagate to the inference server