TTFT: name the metric it captures and what dominates it
TTFT is the wall-clock latency from request submit to first generated token; it is dominated by queue time and prefill, not decode.
Imagine you walked up to a coffee counter and shouted your whole order in one breath. The barista has to listen to every word before she can hand you the first sip. That listening time is the prefill phase, and the wait in line behind other customers is the queue. Picture how long it takes the first drop of coffee to reach your hand: that is TTFT. Whether the rest of the cup fills fast or slow does not change that first-drop moment. Chat apps feel snappy or sluggish almost entirely on that first-drop number, which is why product teams obsess over it more than total cups per minute.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
TTFT, time to first token, is the single most operator-visible latency number on any streaming LLM endpoint. Every chat product, every coding assistant, every voice-mode interface measures it, alerts on it, and ties product SLOs to it. Yet many engineers conflate it with throughput or with total request time, which means they tune the wrong things when latency complaints arrive.
This deep dive walks through what TTFT actually measures, why the two compute phases of LLM inference put prefill on one side of the line and decode on the other, how production stacks instrument it, and which optimizations move it.
What TTFT measures, precisely
TTFT is the wall-clock interval from the moment a client sends a request to the moment the first generated token reaches the client. It includes every cost paid along the way: network ingress, queue wait on the server, prefill compute, the first sampling step, and network egress for that first chunk.
For streaming endpoints, TTFT is identifiable as the time to first SSE delta or the time to first message_start event in Anthropic's API. For non-streaming endpoints the concept still applies as a server-side measurement, but the client only ever sees total request time, which conflates TTFT and the entire decode loop.
What TTFT does NOT include
Decode steps after the first token. Those belong to TPOT (time per output token) and to total request time, but they are paid in the decode loop, not the prefill loop, so they are downstream of the first-token boundary.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Messages API emits a `message_start` SSE event marking exactly when TTFT ends; teams use it to set per-region SLOs on Claude Sonnet 4.6.
- OpenAI's GPT-5.5 streaming endpoint reports TTFT separately in its server logs, and product teams alert on p95 TTFT independent of total tokens emitted.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you reduce p95 TTFT on a long-context chat product without dropping throughput?
Walk through the two axes: prefix caching for repeated system prompts (Anthropic 0.1x read, OpenAI 0.5x on 1024+ token prefixes), then chunked prefill to keep batch step time bounded, then prefill-decode disaggregation so a long prefill does not block decode slots. Mention TP-group sizing as the last resort because it costs throughput.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming TTFT depends on output length. It does not. TTFT is paid before decode begins, and longer outputs only inflate TPOT and total time.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.