Trace how one slow request stalls every co-batched generation step.

Attempt it

On a continuous-batching server, a single pathological request causes p99 latency to spike across all in-flight sequences, even ones that should be unaffected. Explain the mechanism that couples them, name the most common culprits, and describe how you would isolate the slow path so co-batched requests are not held hostage.

Free · 2 AI evals / day

Quick reference

TL;DR

Continuous batching advances every sequence on one shared step thread, so any CPU hook or grammar check that blocks the step holds every co-batched request hostage.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a tour bus that stops at every passenger's chosen station, but only moves when everyone has finished boarding or disembarking. If one passenger fumbles with a heavy suitcase at door, the whole bus waits, even passengers who already sat down. The bus is the GPU step. Each passenger is a request in the batch. The fumble is some slow CPU task that runs between forward passes: turning token IDs into text, checking a complicated grammar, or scoring extra log-probabilities. The driver cannot leave any passenger behind, so one slow passenger sets the speed for everyone. The fix is to move the slow tasks off the bus itself, onto helpers waiting at the curb, and to put a hard timer on each boarding step so a stuck passenger gets dropped rather than freezing the route.

Key concepts

Continuous batching Production debugging Model serving

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: step thread as shared barrier + CPU hooks that block it + structured-output grammar tail + worker-pool isolation + per-step timeout + replica-level workload split.

Real-world examples

Real products, models, and research that use this idea.

vLLM exposes a step-level scheduler where detokenization and logprob hooks run on the step thread; production teams move them to async workers to protect p99.
SGLang serves Llama 4 Maverick with structured-output grammars and dedicates separate replicas for grammar-heavy traffic to keep interactive routes clean.
TensorRT-LLM in-flight batching documents per-step callback budgets explicitly so operators can spot a slow plugin before it stalls the batch.
Anthropic API tiering separates tool-using traffic from simple completion traffic, which is partly a workload-isolation move against the same coupling problem.
DeepSeek V4 reasoning traffic, with its long hidden thinking phase, is often routed to dedicated replicas because reasoning-token bookkeeping can dominate step time.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you instrument a vLLM-like server to confirm the step thread is the bottleneck rather than the GPU?

Sign in or go Pro to see the approach for this follow-up.

QWhy is moving detokenization to a worker pool safe but moving sampling to a worker pool dangerous?

Sign in or go Pro to see the approach for this follow-up.

QHow do per-step timeouts interact with structured-output correctness?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Blaming the GPU when the real culprit is a CPU-side hook running on the step thread. The GPU finishes in milliseconds while a Python detokenizer or grammar check holds the barrier.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.