TTFT: name the metric it captures and what dominates it

**TTFT** stands for Time To First Token. It is the wall-clock gap between the moment a client sends a request and the moment the first generated token arrives back at the client. An LLM completion has two compute phases. The first is prefill, which reads the entire input prompt and runs one big forward pass to populate the KV cache and produce the first logits. The second is decode, which generates one new token per forward pass after that. TTFT measures only the time up to and including that first token, so it is the queue time on the server plus the prefill compute. Decode never enters TTFT. This matters for product teams because users feel responsiveness through the first character they see on screen. With streaming on, the rest of the answer can flow at a moderate rate and still feel fast, but a slow first token feels broken no matter how fast the rest comes. Modern hosted APIs like GPT-5.5 and Claude Sonnet 4.6 report TTFT separately from throughput for exactly this reason.

**TTFT** is the wall-clock latency from request submission to the first generated token reaching the caller. Two things dominate the number: queue time on the server while the request waits for a slot, and prefill compute, which runs one parallel forward pass over the entire input prompt to fill the KV cache and emit the first logits. Decode-step latency is not part of TTFT. That work shows up in TPOT (time per output token) and in total request time. So TTFT scales with input length and load, not with output length. The reason chat product teams obsess over TTFT more than aggregate tokens per second is perceptual. Once streaming is on, the user starts reading at the first token and tolerates a moderate stream rate after. A slow first token feels broken regardless of how fast the rest flows. Anthropic's API reports a `message_start` event marking exactly this boundary; OpenAI exposes it through the first SSE delta. Every serving stack worth its name (vLLM, TensorRT-LLM, SGLang) publishes p50 and p95 TTFT separately from throughput because the two numbers move on different axes.