Match TTFT and TPOT to the latency they measure during streaming
Drag each answer to line up with its matching prompt
TTFT
Prefill: one compute bound forward pass over the whole prompt
TPOT
Decode: one bandwidth bound step per generated token
Phase that dominates TTFT
Time Per Output Token, inter token gap during streaming
Phase that dominates TPOT
Time To First Token, wall clock from request to the first content token
Total perceived latency for an N-token reply
TTFT + N * TPOT
TTFT is wall-clock to the first content token (dominated by prefill); TPOT is the inter-token gap during streaming (dominated by decode). Total latency is TTFT + N * TPOT.
Imagine ordering a coffee. TTFT is how long you wait from the moment you order until the first drop hits the cup. TPOT is how steadily the rest of the coffee streams in after that. Two cafes might both deliver a full cup in thirty seconds, but the one with a five-second TTFT feels much faster than the one that takes twenty-five seconds to start pouring. LLM serving is the same. Users care less about total time than about how quickly something starts arriving and how smoothly the rest follows.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
TTFT and TPOT are the two standard latency metrics for streaming LLM serving. Every production SLO, every benchmark and every optimization argument in LLM inference refers back to these two numbers.
They exist as separate metrics because LLM inference has two phases with opposite cost profiles. The prefill phase is one big compute-bound pass over the entire prompt. The decode phase is many small bandwidth-bound steps, one per generated token. Lumping them into a single end to end latency hides the information you need to debug a slow endpoint.
What TTFT and TPOT measure
TTFT, Time To First Token, is the wall-clock interval between request submission and the arrival of the first content token in the streaming response. It captures everything the user experiences before any text appears.
The key word is content. In Anthropic's streaming protocol, the first SSE event is message_start, a metadata preamble carrying the message ID and model name. Its content is empty. Using message_start as the TTFT marker can understate the user-perceived wait. The correct marker is the first content_block_delta of type text_delta. OpenAI's first chunk often has an empty delta, with the first content arriving in a later chunk; the right marker is the first chunk where choices[0].delta.content is non-empty.
TPOT, Time Per Output Token, is the average inter-token gap during streaming. If a 100-token response takes 5 seconds to stream after TTFT, TPOT is 50 ms. The metric is meaningful because decode is steady: each step has roughly the same wall-clock cost on the same hardware at the same batch size.
Total perceived latency for an N-token reply is TTFT + N * TPOT, plus a small flush at the end. This decomposition is what makes the two metrics actionable: any optimization either cuts the upfront one-time cost or cuts the per-token recurring cost.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Metric | What it measures | Bound by | Scales with |
|---|---|---|---|
| TTFT | Request to first content token | Prefill compute, HBM bandwidth for KV write | Prompt length |
| TPOT | Inter-token gap during streaming | HBM bandwidth for weights and KV reads | Roughly flat per token, slight growth with context |
Real products, models, and research that use this idea.
- DistServe (OSDI 2024) places prefill and decode on different GPUs and measures goodput as the request rate that meets both a TTFT and a TPOT target.
- Anthropic's streaming API sends message_start with empty content before any text, so client TTFT timers should stop at the first content_block_delta.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does prompt caching cut TTFT but not TPOT?
Caching reuses prefill KV state for a byte-identical prefix. That cuts the work of the prefill phase, which only affects TTFT. Decode still has to generate each output token from scratch, so TPOT is unchanged.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating TTFT and TPOT as the same metric, or measuring them at the wrong event. TTFT ends at the first content delta, not the first SSE event.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.