Match TTFT and TPOT to the latency they measure during streaming

Match pairs·Easy·4.0 · 0·~2 min

Attempt it

TTFT

TPOT

Phase that dominates TTFT

Phase that dominates TPOT

Total perceived latency for an N-token reply

**TTFT** and **TPOT** are the two latency metrics every LLM serving team tracks. **TTFT** stands for **Time To First Token**. It measures wall-clock time from the moment the request is sent to the moment the first content token appears in the response stream. This is dominated by the **prefill** phase, where the model reads the entire prompt and computes keys and values for every position. Prefill is one big compute-bound matmul, and its cost scales with prompt length. Long prompts make TTFT long. **TPOT** stands for **Time Per Output Token**. It measures the average gap between consecutive content tokens once streaming starts. This is dominated by the **decode** phase, where the model produces one token per step. Each decode step reads the model weights and the KV cache from HBM, so TPOT is bandwidth-bound and roughly flat per token. The total perceived latency for an N-token reply is **TTFT + N * TPOT**. A chat UI feels snappy when TTFT is low (something appears quickly) and TPOT is steady (the rest streams smoothly). These two metrics get optimized through different levers because they are bound by different phases of inference.

**TTFT** (Time To First Token) and **TPOT** (Time Per Output Token) are the canonical streaming-latency metrics. They separate cleanly because LLM inference has two phases with opposite cost profiles. TTFT is wall-clock from request submission to the arrival of the first content token. It is dominated by prefill, which is one compute-bound matmul over the whole prompt. Prefill cost scales with prompt length, so TTFT grows with input size and saturates the tensor cores once batched. TPOT is the inter-token gap during streaming, the average wait between successive content tokens. It is dominated by decode, which produces one token per step. Each step reads the entire model weight from HBM plus the KV cache, so decode is HBM-bandwidth-bound. TPOT is roughly flat per token, set by `bytes_per_token / bandwidth`. Total latency is approximately TTFT + N * TPOT. The shape of that equation matters: TTFT is a one-time cost paid up front, while TPOT scales linearly with the response length. Different optimizations attack different metrics. Prompt caching, chunked prefill, and shorter prompts cut TTFT. Quantization, GQA, smaller models, and FlashAttention cut TPOT. Batching helps both but in different ways: it raises prefill throughput and lifts decode utilization. Knowing which metric is the bottleneck for your workload tells you which knob to pull.

Quick reference

TL;DR

TTFT is wall-clock to the first content token (dominated by prefill); TPOT is the inter-token gap during streaming (dominated by decode). Total latency is TTFT + N * TPOT.

Memory aid

Easy to grasp

Imagine ordering a coffee. TTFT is how long you wait from the moment you order until the first drop hits the cup. TPOT is how steadily the rest of the coffee streams in after that. Two cafes might both deliver a full cup in thirty seconds, but the one with a five-second TTFT feels much faster than the one that takes twenty-five seconds to start pouring. LLM serving is the same: users do not care about total time as much as how quickly something starts arriving and how smoothly the rest follows.

Key concepts

Time to first token Time per output token Prefill Decode Streaming inference

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: TTFT = first content token, dominated by prefill + TPOT = per-token gap, dominated by decode + total = TTFT + N * TPOT + different optimizations for each + measurement pitfalls.

Side-by-side

Metric	What it measures	Bound by	Scales with
TTFT	Request to first content token	Prefill compute, HBM bandwidth for KV write	Prompt length
TPOT	Inter-token gap during streaming	HBM bandwidth for weights and KV reads	Roughly flat per token, slight growth with context

Real-world examples

Real products, models, and research that use this idea.

OpenAI publishes TTFT and TPOT for GPT-5.5 on its status page, breaking down latency by region and model tier.
Anthropic's Claude Opus 4.7 and Sonnet 4.6 endpoints expose comparable per-token streaming latency to enterprise customers.
vLLM benchmarks report TTFT and TPOT separately for Llama 3.1 70B on H100, illustrating the prefill versus decode split.
TensorRT-LLM MLPerf submissions report median and p99 TPOT to demonstrate steady streaming under load.
DeepSeek V4 latency tuning explicitly splits TTFT goals from TPOT goals when budgeting prompt length and batch size.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does prompt caching cut TTFT but not TPOT?

QHow does speculative decoding affect TPOT?

QWhy does p99 TPOT matter more than p50 for chat UX?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating TTFT and TPOT as the same metric, or measuring them at the wrong event. TTFT ends at the first content delta, not the first SSE event.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.