Tokens Per Second (TPS)
Also known as: TPS, Decode throughput
Output tokens per second per request: the steady-state generation rate.
Output throughput of an LLM, measured in tokens generated per second per request (decode throughput). Inversely proportional to per-token latency once streaming has started.
In practice
Second main latency metric after TTFT. Interviews probe how it changes with batch size, quantization, and speculative decoding.
How it compares
TTFT measures first-token latency; TPS measures sustained generation throughput after the first token.
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.