A serving system measures the following on an H100 with a 7B model: - At batch=1: throughput = 80 tok/s, TPOT (per request) = 12.5 ms/tok - At batch=8: throughput = 600 tok/s, TPOT = 13.3 ms/tok - At batch=32: throughput = 2200 tok/s, TPOT = 14.5 ms/tok - At batch=128: throughput = 6400 tok/s, TPOT = 20 ms/tok - At batch=256: throughput = ? - At batch=512: throughput = ? The critical batch (compute roof) is reached around batch=128. Predict the throughput and TPOT for batch=256 and batch=512, given that the system is now compute bound. Express each throughput as tokens/second and each TPOT in ms/tok.
Past the compute roof throughput plateaus near 7000 tok/s, so batch=256 and batch=512 barely climb, while per-request TPOT grows linearly to roughly 37 ms and 73 ms.
Picture a bus that leaves the stop on a fixed timer no matter how full it is. While seats are empty, adding riders is free: more people move per trip and nobody waits longer. That is the memory-bound phase, where serving more requests per step lifts throughput almost linearly. Once every seat is full, the bus is at capacity. Cramming more people aboard does not move them any faster, the trip still takes the same time, so the count delivered per hour stops rising. That is the compute roof. Worse, the extra riders have to wait for room, so each person's door to door time grows. For an LLM server, adding requests past saturation means each request waits longer for its next token while total throughput barely budges.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Batch size is the single biggest throughput lever in LLM serving, and this question tests whether you understand why the lever stops working past a point. The roofline model gives decode two operating regimes. There is a bandwidth-bound slope where adding batched requests scales throughput almost linearly, and a compute-bound roof where throughput is pinned at peak and latency degrades.
The model says achievable performance is the smaller of two ceilings: what memory bandwidth can feed, and what the compute units can chew. Formally:
Here arithmetic intensity, AI, is FLOPs done per byte moved from HBM. In decode, AI rises with batch because one weight read serves many requests. While AI is below the hardware ridge point you live on the bandwidth term and throughput climbs; once AI passes the ridge you are clamped to the compute term and throughput is flat.
The measured data walks you up the slope from batch=1 to batch=128 and stops at the corner. The task is to identify that corner and project past it. Once you recognise the saturation signature, the numbers fall out of two facts: total tokens per second cannot exceed what the GPU's compute units produce, and a fixed-time step serving more requests makes each request wait longer.
This deep dive verifies the slope from the given points, locates the elbow, projects the two unknown rows, explains the underlying mechanics with a small calculation, separates this decode behaviour from prefill, and ends with the operational rule that production schedulers actually follow.
Why batching is free while you are memory-bound
A decode step has a fixed byte cost dominated by streaming the model weights out of HBM, plus a per-request cost from reading each sequence's KV cache. At batch=1 you pay the full weight read to make a single token, so almost all the bandwidth is spent on overhead rather than useful per-request work. A 7B model in bf16 is roughly 14 GB; pushing 14 GB through HBM to emit one token is the definition of an inefficient step.
Adding more requests to the same step amortises that one weight read across many tokens. The arithmetic intensity, defined as FLOPs done per byte moved, rises roughly in proportion to batch, because the FLOPs scale with the number of requests while the dominant weight bytes stay constant. That is why throughput climbs almost linearly here: 80, then 600, then 2200, then 6400 tokens per second, close to an eighty-fold jump matching the eighty-fold batch increase.
The scaling is not perfectly linear, and the small shortfall is informative. Each batched request brings its own KV-cache reads, so the byte budget grows slowly with batch even though the weight term is shared. Those extra bytes nibble at the bandwidth, which is why 600 at batch=8 is a hair below a clean eight-times-80.
Latency stays nearly flat in this region because the step wall-clock is set by the byte budget, which barely grows when you add a few KV reads. TPOT moving only from 12.5 ms to 14.5 ms across a thirty-two-fold batch increase is the fingerprint of a bandwidth-bound regime. You are getting more work done per step at essentially the same step cost, which is the free lunch that makes batching the highest-leverage knob in serving.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Batch | Throughput (tok/s) | TPOT (ms/tok) | Regime |
|---|---|---|---|
| 1 | 80 | 12.5 | Bandwidth-bound |
| 8 | 600 | 13.3 | Bandwidth-bound |
| 32 | 2200 | 14.5 | Bandwidth-bound |
| 128 | 6400 | 20 | Near compute roof |
| 256 | ~7000 | ~37 | Compute-bound |
| 512 | ~7000-7400 | ~73 | Compute-bound |
Real products, models, and research that use this idea.
- vLLM exposes max_num_seqs (commonly 256 on H100), set just below the critical batch so continuous batching maximizes throughput without wrecking tail latency.
- TensorRT-LLM and Hugging Face TGI both ship max-batch caps in the 128 to 256 range as production defaults for 7B class models.
What an interviewer would ask next. Try answering before peeking at the approach.
QCompute the critical batch for a 7B model on H100 from first principles.
Take the hardware ridge point as peak FLOPs over HBM bandwidth, roughly 989 teraFLOPs over about 3.35 TB per second, giving an arithmetic intensity near 300. Decode intensity scales with batch, so the crossover is where batch reaches that ridge. Add KV-cache bytes to the denominator and the practical elbow drops toward the measured 128.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Extrapolating the early near-linear region forever and predicting batch=256 doubles throughput to about 12800 tok/s. Past the roof throughput is flat; only latency moves.
60 second bullets to scan on the way to the call.
Spot the elbow at batch=128 as the compute crossover from the data
Predict the throughput plateau near peak past the critical batch
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.