Throughput, for an LLM serving system: give the working definition

Attempt it

Throughput is total output tokens produced per second across all concurrent requests on the system, usually reported as tokens/sec or sometimes tokens/sec/GPU. It is the server-side capacity metric: it tells an operator how much work a node can do per unit time, which directly drives cost per token. Latency is a per-request quantity (TTFT, TPOT, end to end), measured from the perspective of one user. The two often trade off: large batches raise throughput by amortizing the weight-load cost across many requests, but each individual request waits longer in the queue and decodes slightly slower. Throughput is what you put on a capacity-planning spreadsheet; latency is what shows up in a user-facing SLO.

Your answer

**Throughput** for an LLM serving system is the total number of output tokens the server produces per second across all the requests it is handling at the same time. It is usually written as `tokens/sec` (sometimes `tokens/sec/GPU` to normalize across cluster sizes). A modern inference engine on an H100 can hit a few thousand tokens per second of aggregate throughput once it batches many concurrent requests together. This is different from latency. Latency is a per-request number: how long one user's reply takes, from request submission to first token (TTFT) or to final token (end to end). Latency is what an end user feels. Throughput is what the operator sees when looking at the GPU's overall productivity. The two metrics trade off. Batching many requests into one forward pass amortizes the cost of reading the model's weights from HBM, which raises the total tokens per second the server can produce. But each individual request in that batch has to wait for the others to finish their step, so its decode is a little slower than it would be alone. As batch size goes up, throughput goes up and per-request latency goes up too. Product teams care about latency because it sets user experience. Operations and finance teams care about throughput because it directly drives cost per token: more tokens out per GPU-hour means lower cost per million tokens served.

**Throughput** is the system-wide token production rate: total output tokens emitted per second, summed across every concurrent request on the server. The standard unit is `tokens/sec` and it is often normalized to `tokens/sec/GPU` so deployments of different sizes can be compared. The number reflects both the underlying hardware capacity and how effectively the serving engine batches and schedules work. Latency is the per-request counterpart. TTFT measures the gap from request submit to first streamed token. TPOT measures the average time between successive tokens during decode. End to end latency adds the two and depends on output length. Latency is scoped to one request; throughput is aggregated across all requests on the GPU. The two metrics trade off, and understanding that tradeoff is central to capacity planning. Batching is the primary mechanism. Continuous batching schedulers (vLLM, TGI, SGLang) pack many in-flight requests into each forward pass, amortizing the HBM-bandwidth cost of streaming the weight matrix on every decode step. As batch size grows: - Throughput rises, because the same weight read serves more requests per cycle. The arithmetic-intensity ratio of compute to memory shifts toward compute, and tensor cores start to saturate. - Per-request latency rises, because each in-flight request waits on the slowest one in the batch at every step, and the queue is longer. The operating point is a business decision. A chat product with a 2-second TTFT SLO caps batch size at whatever keeps p99 TTFT under the line; the resulting throughput is whatever that batch size delivers. A batch-summarization workload with no real-time constraint can run with batch size in the hundreds, maximizing throughput and minimizing cost per token. The OpenAI Batch API charges roughly 50% of real-time precisely because relaxed latency lets the server pack much higher throughput. Throughput is what shows up in cost models.

Quick reference

TL;DR

Throughput is total output tokens per second across all concurrent requests on the server; it drives cost per token and trades off against per-request latency as batch size grows.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a coffee shop. Throughput is how many cups the shop serves per hour across every customer at once. Latency is how long your particular drink takes from order to hand-off. A bigger espresso machine and a larger batch pull raise the shop's hourly capacity but might mean your individual drink waits a bit longer in the queue. An LLM server has the same shape. Throughput counts every output token the server produces per second across all the chats it is handling. Latency is one user staring at their own chat window. Operators and finance teams watch throughput because it sets cost per token. Product designers watch latency because it sets whether the chat feels alive or sluggish. Both metrics are real; they just answer different questions.

Key concepts

Throughput Latency BatchingCost per Token

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define throughput as aggregated output tokens per second across requests, contrast with per-request latency, explain the batching tradeoff and roofline knee, distinguish prefill (compute-bound) from decode (HBM-bound) throughput, and connect to cost per token.

Real-world examples

Real products, models, and research that use this idea.

vLLM on an H100 can hit 4,000+ output tokens/sec aggregate on Llama 3 70B with continuous batching and FP8 weight quantization.
OpenAI's Batch API offers ~50% cheaper pricing because relaxed 24-hour latency lets them run at the throughput-optimal batch size.
Anthropic publishes per-tier rate limits as `tokens/min` (TPM), which is throughput divided across customers; system throughput per backend is much higher than per-customer TPM.
Together AI, Fireworks, and Groq differentiate primarily on tokens/sec/GPU benchmarks for popular open models; Groq's LPU architecture targets very high single-stream tokens/sec at the cost of aggregate throughput.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is decode throughput limited by HBM bandwidth, not tensor-core FLOPS?

Sign in or go Pro to see the approach for this follow-up.

QHow does prompt length distribution affect reported throughput numbers?

Sign in or go Pro to see the approach for this follow-up.

QWhy does the OpenAI Batch API offer roughly 50% discount?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reporting single-stream tokens/sec as system throughput. Single-stream is one request's decode rate; system throughput is summed across every concurrent request on the GPU.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.