TTFT: name the metric it captures and what dominates it
TTFT is the wall-clock latency from request submit to first generated token. Queue time and prefill dominate it, not decode.
Imagine you walked up to a coffee counter and shouted your whole order in one breath. The barista has to listen to every word before she can hand you the first sip. That listening time is the prefill phase, and the wait in line behind other customers is the queue. Picture how long it takes the first drop of coffee to reach your hand: that is TTFT. Whether the rest of the cup fills fast or slow does not change that first-drop moment. Chat apps feel fast or slow mostly because of that first-drop number. That is why product teams care about it more than total cups per minute.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
TTFT, time to first token, is one of the most visible latency numbers on a streaming LLM endpoint. Chat products, coding assistants and voice interfaces measure it, alert on it and tie product SLOs to it. Yet many engineers confuse it with throughput or with total request time, so they tune the wrong things when latency complaints arrive.
What TTFT measures, precisely
TTFT is the wall-clock interval from the moment a client sends a request to the moment the first generated token reaches the client. It includes every cost paid along the way: network ingress, queue wait on the server, prefill compute, the first sampling step, and network egress for that first chunk.
For streaming endpoints, TTFT is the time to the first SSE chunk that carries text. In Anthropic's API that is the first content_block_delta event. The earlier message_start event arrives with empty content. For non-streaming endpoints the concept still applies as a server-side measurement. But the client only sees total request time, which conflates TTFT and the entire decode loop.
What TTFT does NOT include
Decode steps after the first token are not part of TTFT. Those belong to TPOT (time per output token) and to total request time. They are paid in the decode loop, not the prefill loop, so they come after the first-token boundary.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Messages API stream starts with a message_start event whose content is empty. Text arrives in content_block_delta events, so client-side TTFT ends at the first content_block_delta.
- In OpenAI Chat Completions streaming, client-side TTFT ends at the first chunk whose choices[0].delta carries content.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you reduce p95 TTFT on a long-context chat product without dropping throughput?
Walk through the options. Start with prefix caching for repeated system prompts (cache reads at 0.1x input price or less on Anthropic and on OpenAI GPT-5 and later). Then add chunked prefill to keep batch step time bounded, then prefill-decode disaggregation so a long prefill does not block decode slots. Mention TP-group sizing as the last resort because it costs throughput.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming TTFT depends on output length. It does not. TTFT is paid before decode begins, and longer outputs only inflate TPOT and total time.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.