Why capacity-plan an LLM service in tokens per second, not requests per second

Attempt it

Your team sizes an LLM serving cluster by peak requests per second. Explain why tokens per second is the better unit and what requests/sec hides.

Free · 2 AI evals / day

Quick reference

TL;DR

GPU work scales with tokens processed and generated, not with the count of requests — so capacity, autoscaling, and cost must be planned in tokens/sec.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a restaurant kitchen sizing its staff by counting customers walking in. But one customer orders a glass of water and another orders a ten-course banquet. Counting heads tells you nothing about how hard the kitchen has to work. What actually matters is the number of dishes being cooked per minute. For an LLM service, the 'dishes' are tokens — the chunks of text it reads and writes. So you plan capacity by tokens per second, because that's what the GPUs are actually cooking.

Key concepts

LLM System Design Inference Optimization

Quick scan · 2–4 min

Everything important, quickly.

Take 6-8 minutes to lock in the tokens versus requests distinction and the prefill-decode split so you can size any LLM cluster in an interview without reaching for request counts.

Side-by-side

Aspect	Requests/sec	Tokens/sec
Tracks GPU work?	No — a request can be tiny or huge	Yes — maps to actual compute
Handles load variance	Hides it by averaging	Exposes it directly
Useful for autoscaling	Misleading triggers	Accurate triggers
Granularity	One bucket per call	Splits into prefill and decode

Real-world examples

Real products, models, and research that use this idea.

vLLM and TGI report throughput in tokens/sec and expose prefill and decode metrics separately for exactly this reason.
OpenAI and Anthropic price and rate-limit on tokens, not requests, because tokens are what consume GPU capacity.
Autoscaling on serving platforms keys off tokens in flight or KV-cache pressure rather than request count.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QTwo clusters both sustain 10k tokens/sec but one is RAG-heavy and one is chat-heavy. Will the same hardware serve both?

Sign in or go Pro to see the approach for this follow-up.

QHow would you set an autoscaling trigger for an LLM service?

Sign in or go Pro to see the approach for this follow-up.

QWhy do token-heavy requests threaten your tail latency more than the average suggests?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

vLLM documentation