Match TTFT and TPOT to the latency they measure during streaming

Attempt it

TTFT

TPOT

Phase that dominates TTFT

Phase that dominates TPOT

Total perceived latency for an N-token reply

Quick reference

TL;DR

TTFT is wall-clock to the first content token (dominated by prefill); TPOT is the inter-token gap during streaming (dominated by decode). Total latency is TTFT + N * TPOT.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine ordering a coffee. TTFT is how long you wait from the moment you order until the first drop hits the cup. TPOT is how steadily the rest of the coffee streams in after that. Two cafes might both deliver a full cup in thirty seconds, but the one with a five-second TTFT feels much faster than the one that takes twenty-five seconds to start pouring. LLM serving is the same: users do not care about total time as much as how quickly something starts arriving and how smoothly the rest follows.

Key concepts

Time to first token Time per output token Prefill Decode Streaming inference

Quick scan · 2–4 min

Everything important, quickly.

3 min: TTFT = first content token, dominated by prefill + TPOT = per-token gap, dominated by decode + total = TTFT + N * TPOT + different optimizations for each + measurement pitfalls.

Side-by-side

Metric	What it measures	Bound by	Scales with
TTFT	Request to first content token	Prefill compute, HBM bandwidth for KV write	Prompt length
TPOT	Inter-token gap during streaming	HBM bandwidth for weights and KV reads	Roughly flat per token, slight growth with context

Real-world examples

Real products, models, and research that use this idea.

OpenAI publishes TTFT and TPOT for GPT-5.5 on its status page, breaking down latency by region and model tier.
Anthropic's Claude Opus 4.7 and Sonnet 4.6 endpoints expose comparable per-token streaming latency to enterprise customers.
vLLM benchmarks report TTFT and TPOT separately for Llama 3.1 70B on H100, illustrating the prefill versus decode split.
TensorRT-LLM MLPerf submissions report median and p99 TPOT to demonstrate steady streaming under load.
DeepSeek V4 latency tuning explicitly splits TTFT goals from TPOT goals when budgeting prompt length and batch size.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does prompt caching cut TTFT but not TPOT?

Sign in or go Pro to see the approach for this follow-up.

QHow does speculative decoding affect TPOT?

Sign in or go Pro to see the approach for this follow-up.

QWhy does p99 TPOT matter more than p50 for chat UX?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.