Match TTFT and TPOT to the latency they measure during streaming

Match pairs·Easy·4.0 · 0·~2 min·Asked atHcl Meesho Midjourney·Relevant atAnthropic Cloudflare Groq OpenAI

Attempt it

Drag each answer to line up with its matching prompt

TTFT

Prefill: one big compute bound matmul over the whole prompt

TPOT

Decode: one bandwidth bound step per generated token

Phase that dominates TTFT

Time Per Output Token, inter token gap during streaming

Phase that dominates TPOT

Time To First Token, wall clock from request to the first content token

Total perceived latency for an N-token reply

TTFT + N * TPOT

**TTFT** and **TPOT** are the two latency metrics every LLM serving team tracks. **TTFT** stands for **Time To First Token**. It measures wall-clock time from the moment the request is sent to the moment the first content token appears in the response stream. This is dominated by the **prefill** phase, where the model reads the entire prompt and computes keys and values for every position. Prefill is one big compute-bound matmul, and its cost scales with prompt length. Long prompts make TTFT long. **TPOT** stands for **Time Per Output Token**. It measures the average gap between consecutive content tokens once streaming starts. This is dominated by the **decode** phase, where the model produces one token per step. Each decode step reads the model weights and the KV cache from HBM, so TPOT is bandwidth-bound and roughly flat per token. The total perceived latency for an N-token reply is **TTFT + N * TPOT**. A chat UI feels snappy when TTFT is low (something appears quickly) and TPOT is steady (the rest streams smoothly). These two metrics get optimized through different levers because they are bound by different phases of inference.

**TTFT** (Time To First Token) and **TPOT** (Time Per Output Token) are the canonical streaming-latency metrics. They separate cleanly because LLM inference has two phases with opposite cost profiles. TTFT is wall-clock from request submission to the arrival of the first content token. It is dominated by prefill, which is one compute-bound matmul over the whole prompt. Prefill cost scales with prompt length, so TTFT grows with input size and saturates the tensor cores once batched. TPOT is the inter-token gap during streaming, the average wait between successive content tokens. It is dominated by decode, which produces one token per step. Each step reads the entire model weight from HBM plus the KV cache, so decode is HBM-bandwidth-bound. TPOT is roughly flat per token, set by `bytes_per_token / bandwidth`. Total latency is approximately TTFT + N * TPOT. The shape of that equation matters: TTFT is a one-time cost paid up front, while TPOT scales linearly with the response length. Different optimizations attack different metrics. Prompt caching, chunked prefill, and shorter prompts cut TTFT. Quantization, GQA, smaller models, and FlashAttention cut TPOT. Batching helps both but in different ways: it raises prefill throughput and lifts decode utilization. Knowing which metric is the bottleneck for your workload tells you which knob to pull.

TL;DR

TTFT is wall-clock to the first content token (dominated by prefill); TPOT is the inter-token gap during streaming (dominated by decode). Total latency is TTFT + N * TPOT.

Memory aid

Easy to grasp

Imagine ordering a coffee. TTFT is how long you wait from the moment you order until the first drop hits the cup. TPOT is how steadily the rest of the coffee streams in after that. Two cafes might both deliver a full cup in thirty seconds, but the one with a five-second TTFT feels much faster than the one that takes twenty-five seconds to start pouring. LLM serving is the same: users do not care about total time as much as how quickly something starts arriving and how smoothly the rest follows.

Key concepts

Time to first token Time per output token Prefill Decode Streaming inference

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

TTFT and TPOT are the two canonical latency metrics for streaming LLM serving, and the only ones a serving engineer absolutely must internalize. Every production SLO, every benchmark, every optimization argument in the inference world refers back to these two numbers.

The reason they exist as separate metrics is that LLM inference has two phases with opposite cost profiles. The prefill phase is one big compute-bound matmul over the entire prompt. The decode phase is many small bandwidth-bound matmuls, one per generated token. Lumping them into a single end to end latency hides exactly the information you need to debug a slow endpoint.

This deep dive walks through what TTFT and TPOT measure precisely, why the two phases produce different metrics, how to optimize each, the common measurement pitfalls, and how to read benchmark numbers when comparing providers.

What TTFT and TPOT actually measure

TTFT, Time To First Token, is the wall-clock interval between request submission and the arrival of the first content token in the streaming response. It captures everything the user experiences before any text appears.

The key word is content. In Anthropic's streaming protocol, the first SSE event is message_start, a metadata preamble carrying the message ID and model name. It is emitted before prefill produces any text. Using message_start as the TTFT marker understates the user-perceived wait, sometimes by hundreds of milliseconds. The correct marker is the first content_block_delta of type text_delta. OpenAI's first chunk often has an empty delta, with the first content arriving in a later chunk; the right marker is the first chunk where choices[0].delta.content is non-empty.

TPOT, Time Per Output Token, is the average inter-token gap during streaming. If a 100-token response takes 5 seconds to stream after TTFT, TPOT is 50 ms. The metric is meaningful because decode is steady: each step has roughly the same wall-clock cost on the same hardware at the same batch size.

Total perceived latency for an N-token reply is TTFT + N * TPOT, plus a small flush at the end. This decomposition is what makes the two metrics actionable: any optimization either cuts the upfront one-time cost or cuts the per-token recurring cost.

Why the two phases produce different metrics

Which optimizations cut TTFT versus TPOT

Measurement pitfalls and how to read benchmarks

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Metric	What it measures	Bound by	Scales with
TTFT	Request to first content token	Prefill compute, HBM bandwidth for KV write	Prompt length
TPOT	Inter-token gap during streaming	HBM bandwidth for weights and KV reads	Roughly flat per token, slight growth with context

Real products, models, and research that use this idea.

OpenAI publishes TTFT and TPOT for GPT-5.5 on its status page, breaking down latency by region and model tier.
Anthropic's Claude Opus 4.7 and Sonnet 4.6 endpoints expose comparable per-token streaming latency to enterprise customers.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does prompt caching cut TTFT but not TPOT?

Caching reuses prefill KV state for a byte-identical prefix. That cuts the work of the prefill phase, which only affects TTFT. Decode still has to generate each output token from scratch, so TPOT is unchanged.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating TTFT and TPOT as the same metric, or measuring them at the wrong event. TTFT ends at the first content delta, not the first SSE event.

60 second bullets to scan on the way to the call.

Expansion of TTFT and TPOT
Which phase dominates each metric (prefill vs decode)

Primary sources. Browse if you want the original framing.