Spot the errors in this description of continuous batching

**Batching** runs several requests through the GPU together so one expensive read of the model weights is shared across many sequences. The question is when you fix which requests are in the batch. Static batching fixes the batch at admission. All requests start together and the batch runs until the LAST one finishes. The catch: completions have wildly different lengths. A request that wanted 20 tokens sits in a finished, idle slot while a neighbour grinds out 800 tokens. Those idle slots are pure wasted GPU. Continuous batching fixes this by scheduling at the iteration level. After every single decode step, the server checks which requests finished, evicts them, and pulls waiting requests into the freed slots. The batch composition is fluid: requests join and leave constantly. The payoff is throughput. The GPU stays packed with live work instead of holding empty slots behind one long sequence. That is why every modern serving stack, vLLM, TGI, and TensorRT-LLM, makes continuous batching the default rather than an option. The statement in the question inverts all of this: it calls continuous batching static, credits it with reducing TTFT, and confines it to offline APIs. All three are wrong.

**Continuous batching** is iteration-level scheduling, the opposite of the static description given. After every decode step the server evicts finished requests and admits waiting ones into the freed slots. Batch composition changes constantly. Static batching, by contrast, fixes the batch at admission. It runs until the longest sequence in the batch finishes. Completion lengths vary by an order of magnitude. So short requests leave their slots idle while one tail-long sequence drags on. That idle GPU time is the waste continuous batching removes. The primary win is throughput, not TTFT. Iteration-level scheduling keeps every slot doing live work. It does not make any single request start faster. TTFT is governed by prefill scheduling and queue depth, separate concerns. Crucially, continuous batching does NOT pad to the longest sequence. Each request runs exactly its own number of steps. It is the default in online serving, not offline batch. Online chat is precisely where short and long completions intermix. Static batching would waste capacity there catastrophically. vLLM, TGI, and TensorRT-LLM all ship it as the standard scheduler.