On a continuous batching server, a single pathological request causes p99 latency to spike across all in flight sequences, even ones that should be unaffected. Explain the mechanism that couples them, name the most common culprits, and describe how you would isolate the slow path so co-batched requests are not held hostage.
Continuous batching advances every sequence on one shared step thread, so any CPU hook or grammar check that blocks the step holds every co-batched request hostage.
Picture a tour bus that stops at every passenger's chosen station, but only moves when everyone has finished boarding or disembarking. If one passenger fumbles with a heavy suitcase at door, the whole bus waits, even passengers who already sat down. The bus is the GPU step. Each passenger is a request in the batch. The fumble is some slow CPU task that runs between forward passes: turning token IDs into text, checking a complicated grammar, or scoring extra log-probabilities. The driver cannot leave any passenger behind, so one slow passenger sets the speed for everyone. The fix is to move the slow tasks off the bus itself, onto helpers waiting at the curb, and to put a hard timer on each boarding step so a stuck passenger gets dropped rather than freezing the route.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Continuous batching is the throughput primitive that took open-source LLM serving from naive to production: schedule at the token level, evict finished sequences, admit new ones, keep the batch full. The unintended consequence is that every request in the batch now shares a clock. The forward pass advances them together, and the bookkeeping that runs between forward passes is a serialized critical section.
When one request's bookkeeping stalls, every co-batched request pays the same latency. The GPU is usually innocent in these incidents. The step thread, the CPU work between launches, is where the time actually leaks. This deep dive walks through the mechanism, the diagnostic, and the isolation playbook that production teams use to keep p99 honest under continuous batching.
The shape of the problem matters because the temptation, on first sight, is to throw away the throughput win and revert to per-request scheduling. That is rarely the right move. The correct move is to keep continuous batching and surgically remove the work that does not belong on the step thread, with timeouts as a hard backstop and replica-level isolation as a structural backstop.
Why the step thread is a shared lock
Continuous batching needs an internally consistent view of the batch every iteration. The scheduler picks the active sequences, the KV cache reserves their blocks, and the forward pass produces one new token for each. Before the next forward pass can launch, every sequence has to be reconciled: did it hit a stop token, what is its detokenized output for the streaming response, did any constrained-decoding mask need an update.
That reconciliation is serialized by design. You cannot start the next step until you know which sequences are still active and what their masks look like. The step thread is therefore a critical section that holds every batched request. Time spent there is paid by all of them, not amortized across them.
This is the structural reason a single pathological request can spike p99 across the whole batch. The GPU is sized to handle the forward pass on schedule. The step thread is not sized to handle a single slow callback, because the median callback is microseconds. When the tail goes long, the barrier blocks.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM exposes a step-level scheduler where detokenization and logprob hooks run on the step thread; production teams move them to async workers to protect p99.
- SGLang serves Llama 4 Maverick with structured-output grammars and dedicates separate replicas for grammar-heavy traffic to keep interactive routes clean.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you instrument a vLLM-like server to confirm the step thread is the bottleneck rather than the GPU?
Add a per-step timer that splits forward-pass time from post-step CPU time. Correlate p99 step time with which sequences have logprob or grammar hooks enabled. A CPU flame graph on the step thread will show Python frames or grammar-engine frames where you expected CUDA launches.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming the GPU when the real culprit is a CPU-side hook running on the step thread. The GPU finishes in milliseconds while a Python detokenizer or grammar check holds the barrier.
60 second bullets to scan on the way to the call.
Why continuous batching serializes per-step bookkeeping across the batch
What runs on the step thread versus what can run on a worker pool
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.