Your GPU pool sustains 9,000 output tokens/sec at acceptable latency. Traffic is 30 requests/sec, and each request generates on average 250 output tokens. Decide whether the pool has enough headroom, and state the required output-token throughput.
The capacity is quoted in **tokens per second**, but the traffic is given in requests per second. You can't compare them until they're in the same unit.
Convert the traffic to tokens: each request generates about 250 output tokens, and 30 requests arrive per second. So the demand is 30 × 250 = 7,500 output tokens per second.
Now compare. The pool sustains 9,000 tokens/sec and you need 7,500. Since 7,500 is below 9,000, the pool holds.
But don't stop at yes. The pool is running at 7,500 / 9,000 ≈ 83% utilization, leaving only about 17% headroom. That's tight. If a traffic burst hits, or the average answer gets a little longer, you'd cross 9,000 and latency would spike. A safe answer says 'it holds for now, but headroom is thin.'
First, get demand and capacity into the same unit. Capacity is 9,000 output tokens/sec; traffic is 30 req/s at 250 output tokens each. So required throughput is:
$$30 \text{ req/s} \times 250 \text{ tok/req} = 7{,}500 \text{ tok/s}$$
That's below the 9,000 tok/s the pool sustains, so it **holds**. Utilization is 7,500 / 9,000 ≈ 0.83, leaving roughly 17% headroom.
The senior move is to flag that 83% is uncomfortable, not safe. LLM traffic is bursty and output lengths are variable, so a single average hides a heavy tail. A modest burst — say request rate climbing to 36/s — would push demand past 9,000 and blow the latency SLA. The same happens if the average generation grows from 250 to 300 tokens. So the honest answer is that it fits today with thin margin. You'd either add a GPU, set an autoscaling trigger before ~85% utilization, or cap output length to protect the tail.
Quick reference
TL;DR
Multiply 30 req/s by 250 tokens/req to get 7,500 tokens/sec demand against 9,000 capacity — it holds, but 83% utilization leaves thin headroom.
Imagine a kitchen that can plate 9,000 fries a minute. Thirty orders come in each minute, and each order is 250 fries. So the kitchen must plate 30 × 250 = 7,500 fries a minute.
7,500 is under 9,000, so the cooks keep up — but they're running at 83% of their max. One busy minute, or orders that come with extra fries, and they fall behind. The lesson: convert orders into fries before you decide if the kitchen can cope.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Capacity already quoted in requests/sec for a fixed-shape workload If every request has near-identical token counts, the conversion is redundant and requests/sec already tracks load.
Workloads dominated by long prompts and short answers An output-tokens capacity figure misses prefill load, so this decode-only formula understates real demand.
Hard, spiky SLAs where any saturation is unacceptable Mean-utilization math hides queueing delay; you need tail-aware sizing, not a single 83% headroom number.
Quick scan · 2–4 min
Everything important, quickly.
Spend 5-7 minutes on the unit conversion and the headroom reasoning so you can both compute the number and argue why 83% utilization is a problem, not a pass.