In benchmarks, tokens per second measures what exactly?

**Tokens-per-second (T/s)** is how many output tokens get produced per wall-clock second. The catch is that the same engine reports two very different T/s numbers depending on how you aggregate. **Single-stream T/s** is the rate for one isolated request: send one prompt, time how long the decode takes, divide tokens by seconds. For Llama 3.1 70B on an H100, single-stream T/s sits around 50 tokens per second; for a smaller model like Llama 3 8B it can be 100 to 150 tokens per second. This is roughly `1 / TPOT` (time per output token), and it approximates what a user feels in a chat UI: roughly how fast the reply streams in. **System T/s** is the rate summed across all concurrent requests on the server. With continuous batching and 128 in-flight requests on Llama 3.1 70B FP8, the same H100 can hit aggregate system T/s around 3,000 to 4,000. The big number reflects the GPU's total productivity, not one user's experience. System T/s is what drives cost per token and what providers use for capacity planning. Why the difference? Batching. The model has to read its weight matrix from HBM on every decode step regardless of how many requests are in the batch. With one request, that bandwidth serves one user. With 128 requests, the same bandwidth serves 128 users in parallel, so aggregate throughput multiplies even though each individual request decodes at roughly the same speed (with some slowdown). The key skill is reading benchmark numbers carefully. 'Llama 70B at 50 T/s' probably means single-stream. 'Llama 70B at 4,000 T/s' probably means system. Same engine, same GPU, very different numbers, both correct.

**T/s** is the rate of generated tokens per wall-clock second, but the number is meaningless without specifying the aggregation scope. The two canonical definitions are: **Single-stream T/s.** The decode rate of one isolated request running on the GPU with no other concurrent load. Mathematically `1 / TPOT`, where TPOT is the time per output token in steady state. For an H100 serving Llama 3.1 70B in FP8, single-stream T/s typically sits at 45-55 T/s. This is the metric that maps to user-perceived speed: a chat completion streaming at 50 T/s renders at roughly the natural reading rate. **System T/s.** The total output-token production rate of the server, summed across all in-flight requests. On the same H100 with continuous batching and a batch of 64-128 requests, system T/s climbs to 3,000-4,500 for Llama 3.1 70B FP8. This is the metric that drives cost per token and capacity sizing. The gap between these numbers is the batching effect. During decode, each forward pass reads the entire weight matrix from HBM (this is the dominant cost) and uses it to multiply against a tiny per-request activation vector. With batch size 1, that weight read serves one user. With batch size N, the same weight read serves N users in parallel, so aggregate throughput rises sharply. Per-request decode latency rises modestly because each in-flight request waits on the slowest one at every step, but the overall trade is heavily in favor of system throughput. A representative shape for H100 SXM, Llama 3.1 70B FP8: - Batch 1: 50 T/s single-stream, 50 T/s system, ~20ms TPOT. - Batch 16: 45 T/s single-stream, 720 T/s system, ~22ms TPOT. - Batch 64: 38 T/s single-stream, 2,432 T/s system, ~26ms TPOT. - Batch 128: 28 T/s single-stream, 3,584 T/s system, ~36ms TPOT. Reading a benchmark correctly is a survival skill.