Throughput, for an LLM serving system: give the working definition

**Throughput** for an LLM serving system is the total number of output tokens the server produces per second across all the requests it is handling at the same time. It is usually written as `tokens/sec` (sometimes `tokens/sec/GPU` to normalize across cluster sizes). A modern inference engine on an H100 can hit a few thousand tokens per second of aggregate throughput once it batches many concurrent requests together. This is different from latency. Latency is a per-request number: how long one user's reply takes, from request submission to first token (TTFT) or to final token (end to end). Latency is what an end user feels. Throughput is what the operator sees when looking at the GPU's overall productivity. The two metrics trade off. Batching many requests into one forward pass amortizes the cost of reading the model's weights from HBM, which raises the total tokens per second the server can produce. But each individual request in that batch has to wait for the others to finish their step, so its decode is a little slower than it would be alone. As batch size goes up, throughput goes up and per-request latency goes up too. Product teams care about latency because it sets user experience. Operations and finance teams care about throughput because it directly drives cost per token: more tokens out per GPU-hour means lower cost per million tokens served.

**Throughput** is the system-wide token production rate: total output tokens emitted per second, summed across every concurrent request on the server. The standard unit is `tokens/sec` and it is often normalized to `tokens/sec/GPU` so deployments of different sizes can be compared. The number reflects both the underlying hardware capacity and how effectively the serving engine batches and schedules work. Latency is the per-request counterpart. TTFT measures the gap from request submit to first streamed token. TPOT measures the average time between successive tokens during decode. End to end latency adds the two and depends on output length. Latency is scoped to one request; throughput is aggregated across all requests on the GPU. The two metrics trade off, and understanding that tradeoff is central to capacity planning. Batching is the primary mechanism. Continuous batching schedulers (vLLM, TGI, SGLang) pack many in-flight requests into each forward pass, amortizing the HBM-bandwidth cost of streaming the weight matrix on every decode step. As batch size grows: - Throughput rises, because the same weight read serves more requests per cycle. The arithmetic-intensity ratio of compute to memory shifts toward compute, and tensor cores start to saturate. - Per-request latency rises, because each in-flight request waits on the slowest one in the batch at every step, and the queue is longer. The operating point is a business decision. A chat product with a 2-second TTFT SLO caps batch size at whatever keeps p99 TTFT under the line; the resulting throughput is whatever that batch size delivers. A batch-summarization workload with no real-time constraint can run with batch size in the hundreds, maximizing throughput and minimizing cost per token. The OpenAI Batch API charges roughly 50% of real-time precisely because relaxed latency lets the server pack much higher throughput. Throughput is what shows up in cost models.