TTFT: name the metric it captures and what dominates it

Attempt it

Time To First Token is the wall-clock latency from request submission to the first generated token reaching the client. The number is dominated by two things: queue time on the server and the prefill phase, which processes the entire input prompt in one forward pass before any token can be emitted. Decode-step latency does not enter TTFT, only TPOT and total time. Chat product teams care about it more than aggregate tokens per second because perceived responsiveness collapses to TTFT once streaming is on: the user starts reading at the first token and tolerates a moderate stream rate after, but a slow first token feels broken regardless of how fast the rest flows.

Your answer

Quick reference

TL;DR

TTFT is the wall-clock latency from request submit to first generated token; it is dominated by queue time and prefill, not decode.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you walked up to a coffee counter and shouted your whole order in one breath. The barista has to listen to every word before she can hand you the first sip. That listening time is the prefill phase, and the wait in line behind other customers is the queue. Picture how long it takes the first drop of coffee to reach your hand: that is TTFT. Whether the rest of the cup fills fast or slow does not change that first-drop moment. Chat apps feel snappy or sluggish almost entirely on that first-drop number, which is why product teams obsess over it more than total cups per minute.

Key concepts

Time to first token Prefill Decode Time per output token Streaming inference

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: definition + prefill vs decode split + why product cares about TTFT separately + caching and disaggregation as the two big levers + queue depth as the real p95 villain.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's Messages API emits a `message_start` SSE event marking exactly when TTFT ends; teams use it to set per-region SLOs on Claude Sonnet 4.6.
OpenAI's GPT-5.5 streaming endpoint reports TTFT separately in its server logs, and product teams alert on p95 TTFT independent of total tokens emitted.
vLLM v1 and SGLang publish p50/p95/p99 TTFT in their benchmark scripts, separating queue from prefill so operators can tune batch admission policies.
Llama 4 Maverick deployments on H100 fleets use chunked prefill and prefix caching to keep p95 TTFT under one second for long-context chat workloads.
Gemini 3.1 Pro exposes streaming chunks the same way; latency dashboards for Google's AI Studio chart TTFT alongside completion-tokens per second.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you reduce p95 TTFT on a long-context chat product without dropping throughput?

Sign in or go Pro to see the approach for this follow-up.

QWhy does speculative decoding help TPOT but not TTFT?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes about TTFT when the request fits entirely in a cached prefix?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming TTFT depends on output length. It does not. TTFT is paid before decode begins, and longer outputs only inflate TPOT and total time.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.