Why capacity-plan an LLM service in tokens per second, not requests per second

When an LLM serves a request, almost all the GPU work goes into reading the input tokens and generating the output tokens. The request itself — the bare fact that someone called your service — costs almost nothing. That's why **requests per second** is a misleading unit. One request might be a 30-token 'is this spam?' check. Another might read a 4,000-token document and stream back a 1,000-token summary. Both count as one request, but the second does 150 times more work. If you size your cluster by requests/sec, you're averaging over that huge variance. You'll over-provision when traffic is light on tokens, or fall over when it's heavy. Tokens per second tracks the real load — the actual work the GPUs do every second. So that's the unit you use for capacity, autoscaling triggers, and cost estimates.

An LLM request's GPU cost is dominated by token count, not by the request arriving. Reading input and generating output is the work; the request envelope is free. So **requests per second** hides enormous variance. A 30-token classification reply costs almost nothing. A request that ingests a 4k-token document then streams a 1k-token answer is orders of magnitude heavier, yet both count as one request. Plan by requests/sec and you'll either over-provision for light traffic or get crushed when token-heavy requests cluster. Tokens per second maps directly to the work the GPUs do, so it's the unit for capacity sizing, autoscaling triggers, and cost. The refinement that separates a strong answer: split it into prefill tokens (the prompt, processed in one parallel pass) and decode tokens (generated one at a time, memory bandwidth bound). They stress the GPU differently, so a single tokens/sec number still under-describes load unless you track both.