Tokens Per Second (TPS)
Also known as: TPS, Decode throughput
Output tokens per second per request: the steady-state generation rate.
Output throughput of an LLM, measured in tokens generated per second per request (decode throughput). Inversely proportional to per-token latency once streaming has started.
In practice
Second main latency metric after TTFT. Interviews probe how it changes with batch size, quantization, and speculative decoding.
How it compares
TTFT measures first-token latency; TPS measures sustained generation throughput after the first token.
Related topics
Questions that mention this term
Related terms
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.