A vLLM-style serving pod shows healthy GPU memory at startup. Over six to twelve hours, KV-cache utilization climbs monotonically, the scheduler admits fewer concurrent requests, latency rises, and eventually the pod OOMs. Request volume is steady throughout. Diagnose the pattern: what is leaking, what is the upstream trigger, and what three mitigations would you put in place?
Free · 2 AI evals / day
**The leak is in KV cache pages tied to abandoned client streams.** Here is what happens in slow motion.
Each inference request reserves a chunk of GPU memory for its KV cache. The reservation is held until the request completes normally (EOS token) or is explicitly cancelled. The serving framework, like vLLM or TensorRT-LLM, returns the pages to a free pool at one of those two events.
When a user closes a browser tab, kills the network on their phone, or a load balancer times out a long SSE connection, no cancel signal reaches the inference engine. The request is, from the engine's perspective, still alive and still streaming.
Over six to twelve hours of normal traffic, a small percentage of these orphans accumulate. The free pool shrinks. The scheduler can admit fewer concurrent requests, so latency climbs even though traffic is steady. Eventually the next admission cannot find enough free pages and the pod OOMs.
The fix is three independent defenses: an idle-sequence reaper that frees sequences with no token progress for N seconds, a hard max session age cap, and cancel propagation from the gateway down to the inference API.
**Diagnosis.** The leaking entity is the per-sequence KV cache. Pages are held until the scheduler sees EOS or a cancel call. **The upstream trigger is broken client streams** where the gateway, load balancer, or browser tear-down does not propagate to the inference server.
The symptom signature is distinctive. Memory grows monotonically at steady traffic. Free-page count drops over hours. Admitted-concurrency falls in lockstep, so throughput drops and queue depth rises even though incoming RPS is flat. The pod OOMs at the moment the next admission cannot allocate. Reproducible by running a load test that opens streaming connections and then drops them at random.
Three mitigations, in priority order:
1. **Cancel propagation**. Wire the HTTP or SSE disconnect event from the gateway and load balancer through to the inference server's cancel endpoint. This is the root-cause fix; the others are safety nets.
2. **Idle-sequence reaper**. A watchdog that scans active sequences for ones with no token-generation progress in 30-60 seconds and force-frees them. Catches the failure case where the cancel propagation also fails.
3. **Hard max session age**. A flat cap (e.g., 5 minutes) past which any sequence is killed regardless of progress. Catches truly stuck sequences, runaway agent loops, and malicious clients holding TCP alive.
The bug usually lives in the gateway to inference gap. The gateway knows the client is gone within seconds. The inference engine never finds out unless someone explicitly tells it.
Quick reference
TL;DR
Orphaned sequences from abandoned client streams pin KV cache pages forever. Fix with idle reaper, max session age, and end to end cancel propagation from gateway to inference server.
Imagine a coat-check at a busy restaurant. Each guest takes a numbered hanger when they arrive and returns it on the way out. If guests slip out a back door without claiming their coat, the hangers fill up. After enough nights, the rack has no free hangers and new guests get turned away even though the dining room is empty. The KV cache works the same way. Each chat request rents some pages of GPU memory to store its conversation state. The pages come back when the request ends cleanly. But if the user closes the browser tab, the inference engine never gets the end signal, the pages stay rented forever, and the rack slowly fills up. After hours of small leaks, there is no free space left and the pod crashes.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Monotonic memory growth with no request traffic If memory grows with no in-flight requests, the leak is in the framework or kernel level rather than per-sequence KV pages.
Sudden memory jumps tied to specific request sizes Sudden jumps point to long-context requests legitimately consuming cache, not orphaned-sequence leakage.
Memory pattern resets cleanly on cluster restart Restart-fixed leaks could be framework state corruption rather than the KV-page leak pattern, which would still leak on a fresh pod.
Agent platforms where 30-minute streaming is expected An aggressive max session age would kill legitimate long-running agent loops; here the idle reaper with token-progress detection is the safer mitigation.
Quick scan · 2–4 min
Everything important, quickly.
5 min: identify orphaned sequences as the leaking entity, trace the disconnect from browser to inference server, walk through the three-layer mitigation (cancel propagation, idle reaper, max session age), and call out the slowloris variant only the hard cap catches.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's Claude Opus 4.7 serving fleet wires Envoy SSE disconnect events directly to the inference cancel API to prevent this exact failure.
2vLLM 0.6+ ships a built-in finished-flag watchdog that catches sequences with no progress for a configurable interval.
3Together AI's Llama 4 Maverick serving stack uses a 5-minute hard ceiling across all requests on H200 clusters.
4Modal's serverless GPU containers observed this pattern on long-context Claude Sonnet 4.6 deployments and added max session age as a default.
5OpenAI's GPT-5.5 streaming endpoints close stale connections at the gateway and propagate cancellations via a shared request-id channel.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is cancel propagation surprisingly hard to get right across a modern service mesh?
Sign in or go Pro to see the approach for this follow-up.
QHow does the slowloris pattern apply to LLM streaming endpoints?
Sign in or go Pro to see the approach for this follow-up.
QWhy is increasing the KV cache pool size not a real fix?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Blaming the model or scheduler. The leak almost always lives in the gateway to inference cancel path, where client disconnects are silently dropped.