A team's P99 TTFT regressed from 400ms to 1.8s overnight while the model and code are unchanged. Describe the observability you'd want and how you'd use it to localize the regression to a root cause. List the specific metrics and traces involved.
Free · 2 AI evals / day
**Time to first token**, or TTFT, is the delay between a request arriving and the first output token streaming back. It is the latency a user feels most when a chat reply starts. When P99 TTFT jumps overnight, the first job is to figure out which part of the pipeline got slow.
TTFT is the sum of two phases. First, the request waits in a queue until the scheduler admits it onto the GPU. Second, the model runs prefill, the parallel forward pass over the whole prompt that produces token one. If you only log the total, you cannot tell these apart.
The fix is instrumentation. Stamp every request with timestamps at arrival, at admission, and at first token. The gap from arrival to admission is queue wait. The gap from admission to first token is prefill duration. Emit both as per-request data.
Now you can localize. If queue wait exploded, the scheduler is starved, usually because KV cache memory is full and requests cannot get in. If prefill exploded, either prompts got longer or a GPU slowed down. Each branch points at a different metric, so you stop guessing and start measuring.
**TTFT** decomposes into two additive spans. Queue wait is arrival to admission. Prefill is admission to first token. Instrument both per request and pull the P99 of each separately before doing anything else.
This split tells you the branch. A queue-wait blowup means the scheduler cannot admit requests. Read per-batch KV cache utilization, preemption count per second, and arrival rate versus admission rate. High KV utilization plus rising preemptions means memory pressure is starving admission. A traffic spike shows up as arrival rate outrunning admission.
A prefill blowup has two usual causes. Either prompts grew, which you confirm by tracking the P99 of prompt token count over time. Or a GPU degraded, which shows in per-GPU counters: HBM bandwidth utilization and tensor-core occupancy. One outlier card flags a degraded device or a noisy neighbor sharing the host.
The enabling discipline is correlation. Emit OpenTelemetry style spans on every request keyed by request id, scrape scheduler state into Prometheus, and join the two. That maps a slow P99 request directly to a workload change or an infrastructure change, rather than eyeballing dashboards and guessing.
Quick reference
TL;DR
Split TTFT into queue_wait plus prefill, instrument each as a per-request span, then read scheduler state for queue blowups and per-GPU counters plus prompt-length drift for prefill blowups.
Imagine a restaurant where the wait for your first bite is the number you care about. That wait has two parts: how long you stood in line before a table opened, and how long the kitchen took once you sat down. If first-bite time doubled overnight, you must know which part grew. Maybe the line got long because the dining room is full and nobody leaves. Maybe cooking got slow because one stove broke, or because everyone suddenly ordered huge meals. A serving system is the same. You stamp every order with a clock at each step, so you can see whether the line or the kitchen caused the slowdown. Without those stamps you are only guessing which half of the wait went wrong.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-user local inference on one GPU with no scheduler or batching layer There is no queue and no admission contention, so per-batch scheduler metrics add no signal.
A regression in inter-token latency rather than time to first token TPOT lives in the decode loop and HBM bandwidth, so queue and prefill spans are the wrong lens.
Tiny low-traffic deployment where instrumentation overhead dwarfs the requests served Full tracing and Prometheus scraping cost more engineering than the latency they would recover.
Offline batch generation where end to end throughput matters and first-token latency does not Nobody waits on a streaming token, so optimizing TTFT spans is effort spent on the wrong metric.
Quick scan · 2–4 min
Everything important, quickly.
5 min: decompose TTFT into queue wait plus prefill, list scheduler metrics for the queue branch and per-GPU plus prompt-length metrics for the prefill branch, then explain request-id correlation across OpenTelemetry spans and Prometheus.
Real-world examples
Real products, models, and research that use this idea.
1vLLM exposes Prometheus metrics for KV cache usage, running and waiting request counts, and preemptions, the exact gauges you read to localize a queue-wait regression.
2NVIDIA TensorRT-LLM with the Triton backend emits per-request and per-batch latency stats plus GPU utilization counters for prefill attribution.
3OpenTelemetry GenAI semantic conventions standardize span attributes like prompt token count and model id, letting traces from Llama 4 or Claude Opus 4.7 serving join cleanly.
4Grafana dashboards backed by Prometheus and the DCGM exporter surface per-GPU HBM bandwidth and tensor-core occupancy so an outlier card stands out instantly.
5SGLang reports scheduler queue depth and cache hit rate, which distinguishes a prefix-cache miss storm from a genuine prompt-length shift.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you distinguish a degraded GPU from a noisy-neighbor process on a shared host?
Sign in or go Pro to see the approach for this follow-up.
QTTFT is fine but P99 TPOT regressed. Which metrics change and why?
Sign in or go Pro to see the approach for this follow-up.
QWhy is P99 the right statistic here rather than the mean or P50?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating TTFT as one opaque number. Without splitting queue wait from prefill, you cannot tell a scheduler admission problem from a slow prefill, so every fix is a guess.