A GPU pool sustains 9,000 output tokens/sec: will it hold this traffic?
Your GPU pool sustains 9,000 output tokens/sec at acceptable latency. Traffic is 30 requests/sec, and each request generates on average 250 output tokens. Decide whether the pool has enough headroom, and state the required output token throughput.
Multiply 30 req/s by 250 tokens/req to get 7,500 tokens/sec demand against 9,000 capacity — it holds, but 83% utilization leaves thin headroom.
Imagine a kitchen that can plate 9,000 fries a minute. Thirty orders come in each minute, and each order is 250 fries. So the kitchen must plate 30 × 250 = 7,500 fries a minute. 7,500 is under 9,000, so the cooks keep up — but they're running at 83% of their max. One busy minute, or orders that come with extra fries, and they fall behind. The lesson: convert orders into fries before you decide if the kitchen can cope.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This looks like a one-line arithmetic problem, and the arithmetic really is one line. But it's a hard question because the trap isn't the math — it's the instinct to compare the two numbers you're handed without noticing they're in different units, and the instinct to stop at 'yes, it fits' without asking how comfortably.
The scenario hands you a request rate, an average output length, and a capacity in tokens. A weak answer divides or compares the wrong pair and either over-builds or, worse, signs off on a pool that's about to fall over. A strong answer does the conversion deliberately, states the utilization, and then reasons about whether that utilization is safe given how LLM traffic actually behaves.
What follows walks through the conversion, the headroom interpretation, why a single average output length is a dangerous simplification, and what the '9,000 output tokens/sec' figure quietly leaves out. The goal is to turn a calculator exercise into the kind of capacity reasoning a senior engineer does out loud in a design review.
Step one: get everything into tokens per second
The capacity you're given is 9,000 output tokens per second — the rate at which the pool can generate text at acceptable latency. The traffic is described in two numbers that aren't directly comparable to that: 30 requests per second, and 250 output tokens per request on average.
The only correct move is to convert the traffic into the capacity's unit. Each request produces about 250 output tokens, and 30 arrive every second, so the demand the pool must satisfy is:
Now both sides speak tokens per second: 7,500 demanded, 9,000 available. The comparison is finally apples to apples.
This sounds trivial, but the whole question exists because the natural instinct is to anchor on '30 requests/sec' — a small, friendly number — and never convert it. Once you've internalized that GPU capacity is spent on tokens, the conversion becomes automatic. Until then, it's the single step that separates a right answer from a confidently wrong one.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
req_per_sec = 30
avg_output_tokens = 250
capacity_tok_per_sec = 9000
demand = req_per_sec * avg_output_tokens # 7500 tok/s
utilization = demand / capacity # 0.833
holds = demand <= capacity # True
print(demand, round(utilization, 2), holds) # 7500 0.83 TrueReal products, models, and research that use this idea.
- vLLM and TGI report sustained output tokens/sec, the exact capacity figure this scenario uses.
- Autoscalers on serving platforms trigger on tokens in flight or KV-cache occupancy rather than request count.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf average output grows from 250 to 320 tokens, does the pool still hold?
Recompute demand as 30 × 320 = 9,600 tok/s, compare to 9,000, and conclude it now exceeds capacity and needs scale-out.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Comparing 30 requests/sec to a 9,000 tokens/sec capacity directly without converting requests into token demand first.
60 second bullets to scan on the way to the call.
Converting requests/sec and tokens/request into a single token-demand figure
Computing utilization as demand divided by capacity
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.