Time to First Token (TTFT)
Also known as: TTFT
Latency from request to the first token landing in the client; the dominant chat UX latency metric.
Latency from request submission to the first generated token reaching the client. Driven by prefill compute (proportional to prompt length) plus queue wait. The dominant perceived latency metric for chat apps.
In practice
First number any production LLM team tracks. Interviews probe prefill vs decode, batching impact, and prompt-caching gains.
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.