What traces and metrics do you need to debug a P99 TTFT regression in production LLM serving?

**Time to first token**, or TTFT, is the delay between a request arriving and the first output token streaming back. It is the latency a user feels most when a chat reply starts. When P99 TTFT jumps overnight, the first job is to figure out which part of the pipeline got slow. TTFT is the sum of two phases. First, the request waits in a queue until the scheduler admits it onto the GPU. Second, the model runs prefill, the parallel forward pass over the whole prompt that produces token one. If you only log the total, you cannot tell these apart. The fix is instrumentation. Stamp every request with timestamps at arrival, at admission, and at first token. The gap from arrival to admission is queue wait. The gap from admission to first token is prefill duration. Emit both as per-request data. Now you can localize. If queue wait exploded, the scheduler is starved, usually because KV cache memory is full and requests cannot get in. If prefill exploded, either prompts got longer or a GPU slowed down. Each branch points at a different metric, so you stop guessing and start measuring.

**TTFT** decomposes into two additive spans. Queue wait is arrival to admission. Prefill is admission to first token. Instrument both per request and pull the P99 of each separately before doing anything else. This split tells you the branch. A queue-wait blowup means the scheduler cannot admit requests. Read per-batch KV cache utilization, preemption count per second, and arrival rate versus admission rate. High KV utilization plus rising preemptions means memory pressure is starving admission. A traffic spike shows up as arrival rate outrunning admission. A prefill blowup has two usual causes. Either prompts grew, which you confirm by tracking the P99 of prompt token count over time. Or a GPU degraded, which shows in per-GPU counters: HBM bandwidth utilization and tensor-core occupancy. One outlier card flags a degraded device or a noisy neighbor sharing the host. The enabling discipline is correlation. Emit OpenTelemetry style spans on every request keyed by request id, scrape scheduler state into Prometheus, and join the two. That maps a slow P99 request directly to a workload change or an infrastructure change, rather than eyeballing dashboards and guessing.