T/s is generated tokens per wall-clock second, but the number depends on how it is aggregated. Single-stream T/s is the rate for one isolated request (e.g. 50 T/s on a single Llama 3.1 70B decode), which approximates user-perceived speed and is roughly 1 / TPOT. System T/s is the rate summed across all concurrent requests on the server (e.g. 4,000 T/s with 128 in-flight requests), which captures server capacity and is what cost per token is computed from. Same engine, same GPU, very different numbers, because batching trades single-stream speed for aggregate throughput. Always check which one a benchmark is reporting before comparing systems.
**Tokens-per-second (T/s)** is how many output tokens get produced per wall-clock second. The catch is that the same engine reports two very different T/s numbers depending on how you aggregate.
**Single-stream T/s** is the rate for one isolated request: send one prompt, time how long the decode takes, divide tokens by seconds. For Llama 3.1 70B on an H100, single-stream T/s sits around 50 tokens per second; for a smaller model like Llama 3 8B it can be 100 to 150 tokens per second. This is roughly `1 / TPOT` (time per output token), and it approximates what a user feels in a chat UI: roughly how fast the reply streams in.
**System T/s** is the rate summed across all concurrent requests on the server. With continuous batching and 128 in-flight requests on Llama 3.1 70B FP8, the same H100 can hit aggregate system T/s around 3,000 to 4,000. The big number reflects the GPU's total productivity, not one user's experience. System T/s is what drives cost per token and what providers use for capacity planning.
Why the difference? Batching. The model has to read its weight matrix from HBM on every decode step regardless of how many requests are in the batch. With one request, that bandwidth serves one user. With 128 requests, the same bandwidth serves 128 users in parallel, so aggregate throughput multiplies even though each individual request decodes at roughly the same speed (with some slowdown).
The key skill is reading benchmark numbers carefully. 'Llama 70B at 50 T/s' probably means single-stream. 'Llama 70B at 4,000 T/s' probably means system. Same engine, same GPU, very different numbers, both correct.
**T/s** is the rate of generated tokens per wall-clock second, but the number is meaningless without specifying the aggregation scope. The two canonical definitions are:
**Single-stream T/s.** The decode rate of one isolated request running on the GPU with no other concurrent load. Mathematically `1 / TPOT`, where TPOT is the time per output token in steady state. For an H100 serving Llama 3.1 70B in FP8, single-stream T/s typically sits at 45-55 T/s. This is the metric that maps to user-perceived speed: a chat completion streaming at 50 T/s renders at roughly the natural reading rate.
**System T/s.** The total output-token production rate of the server, summed across all in-flight requests. On the same H100 with continuous batching and a batch of 64-128 requests, system T/s climbs to 3,000-4,500 for Llama 3.1 70B FP8. This is the metric that drives cost per token and capacity sizing.
The gap between these numbers is the batching effect. During decode, each forward pass reads the entire weight matrix from HBM (this is the dominant cost) and uses it to multiply against a tiny per-request activation vector. With batch size 1, that weight read serves one user. With batch size N, the same weight read serves N users in parallel, so aggregate throughput rises sharply. Per-request decode latency rises modestly because each in-flight request waits on the slowest one at every step, but the overall trade is heavily in favor of system throughput.
A representative shape for H100 SXM, Llama 3.1 70B FP8:
- Batch 1: 50 T/s single-stream, 50 T/s system, ~20ms TPOT.
- Batch 16: 45 T/s single-stream, 720 T/s system, ~22ms TPOT.
- Batch 64: 38 T/s single-stream, 2,432 T/s system, ~26ms TPOT.
- Batch 128: 28 T/s single-stream, 3,584 T/s system, ~36ms TPOT.
Reading a benchmark correctly is a survival skill.
Quick reference
TL;DR
T/s is generated tokens per second, but it splits into two metrics: single-stream T/s (one request's decode rate, user-facing) and system T/s (aggregate across all in-flight requests, capacity-facing).
Picture a chef plating dishes. Single-stream T/s asks how many bites this one diner gets per second. System T/s asks how many bites the whole restaurant serves per second across every diner at once. With one diner, both numbers are the same. With sixty-four diners, the kitchen sends out hundreds of bites per second across the room, even though any one diner is still getting bites at roughly their normal pace, maybe a bit slower. So if a vendor brags '4,000 T/s on H100', they are probably counting the whole restaurant. If your chat reply feels slow at 30 T/s, you are measuring your one plate. Both are real; they just answer different questions, and you have to know which the benchmark is reporting.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You're comparing inference systems for a real-time chat product. Single-stream T/s and TPOT under realistic batch load are what matter; aggregate system T/s misleads because it does not reflect what users feel.
You're sizing capacity for a strict latency SLO. Peak system T/s at unbounded batch is not achievable when you have a TPOT cap; report throughput at the SLO-bounded batch size instead.
You're comparing two different model architectures. T/s does not normalize for work per token; a larger model at the same T/s is doing more compute per token, which raw T/s does not capture.
Quick scan · 2–4 min
Everything important, quickly.
5 min: define T/s as tokens per wall-clock second, distinguish single-stream from system T/s, explain why batching makes them diverge, give H100 numbers for both at low and high batch sizes, and name one canonical benchmarking tool.
Real-world examples
Real products, models, and research that use this idea.
1Groq's LPU architecture publishes single-stream T/s benchmarks like '300 T/s on Llama 3 70B', targeting per-request speed at the cost of aggregate throughput.
2vLLM and TGI on H100 publish system T/s benchmarks like '4,000 T/s on Llama 3 70B' at batch sizes of 128 in-flight requests.
3NVIDIA's genai-perf tool reports both single-stream T/s and system T/s as a function of concurrency level on the same plot.
4Together AI, Fireworks, and Anyscale leaderboards compare hosted Llama inference primarily on system T/s/GPU at standardized batch sizes.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does single-stream T/s on a 70B model max out around 50 T/s on H100?
Sign in or go Pro to see the approach for this follow-up.
QHow does Groq's LPU achieve 300+ T/s single-stream when H100 maxes around 50?
Sign in or go Pro to see the approach for this follow-up.
QWhat's the difference between TPS, TPM, and 'system throughput' on a dashboard?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Comparing one vendor's single-stream T/s to another's system T/s without normalizing. Both numbers are called 'T/s' and they are not the same thing.