Drag each answer to line up with its matching prompt
TTFT (time to first token)
Delay from request to the first streamed token; driven by queue wait + prefill
TPOT (time per output token)
The tail: 99% of requests finish at or under this time
End to end latency
Total time to the complete response; ≈ TTFT + TPOT × output tokens
p99 latency
Aggregate token generation rate across all concurrent requests
Throughput (tokens/sec)
Average gap between successive streamed tokens; sets streaming speed
TTFT is the wait for the first token (queue + prefill), TPOT is the per-token streaming gap, and end to end ≈ TTFT + TPOT × output length.
Imagine ordering food at a counter. There's the wait until your first plate arrives, and then how quickly each following plate comes out. Those are two separate experiences. LLM latency works the same way. TTFT is how long until the first word shows up. TPOT is how fast the rest of the words keep coming after that. The total time you wait is the first-word wait plus the per-word pace times how many words there are. The kitchen's overall output across all customers is throughput.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Latency for an LLM is not one number, and treating it as one leaves the question "fast by which measure?" unanswered. A streamed completion unfolds over hundreds of tokens. A single round-trip figure throws away the information you need to debug or to write an SLA.
The useful framing is two axes. First, per-request versus fleet-aggregate: some metrics describe what one user feels, others describe what the system can sustain. Second, central tendency versus tail: an average and a p99 can differ widely for the same service.
Per-request startup: TTFT
TTFT, time to first token, is the delay from when a request arrives to when the first token streams back. It is the metric a user experiences as raw responsiveness: the gap between hitting enter and seeing anything happen.
What drives it is queue wait plus prefill. The request first sits in the scheduler's queue until the serving loop can admit it. Then the model runs a prefill pass over the entire prompt before it can emit token one. So TTFT grows with system load and prompt length. It is independent of how long the eventual answer will be.
This independence is the point of streaming. Without streaming, the user waits the full end to end time before seeing a single character. With streaming, they wait only TTFT (often a few hundred milliseconds), and then text flows. A serving team watches TTFT closely because it is the first impression. Prompt caching, smarter queueing, and prefill optimization all move it directly without touching the rest of the pipeline.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Metric | Scope | Driven by |
|---|---|---|
| TTFT | Per request, startup | Queue wait + prefill |
| TPOT | Per request, streaming | Decode rate per token |
| End to end | Per request, total | TTFT + TPOT × output tokens |
| p99 latency | Per request, tail | Distribution of end to end times |
| Throughput | Fleet aggregate | Concurrent tokens/sec across all requests |
Real products, models, and research that use this idea.
- Teams measure TTFT and inter-token latency on the OpenAI and Anthropic streaming chat APIs.
- vLLM's benchmark suite reports TTFT, TPOT, and aggregate throughput side by side.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can raising throughput via batching worsen TTFT and p99?
Tie larger batches to added scheduling and queue delay that lengthen startup and the tail even as tokens/sec rises.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Conflating per-request latency with throughput, or assuming a high tokens/sec number means low latency. The two can move in opposite directions.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.