Which metrics are essential to debug a P99 TTFT regression in production LLM serving?

**Time to first token** (TTFT) is how long a user waits from sending a request to seeing the first output word. A P99 regression means the slowest 1 percent of requests got worse, even if the typical request is fine. TTFT breaks into two parts you can measure per request. First, queue wait: time the request sits waiting for the scheduler to admit it into a batch. Second, prefill: time the model spends reading the whole prompt in one forward pass before it can emit token one. To debug, you want a metric for each part. Queue wait and prefill duration, captured per individual request, tell you which half regressed. That alone narrows the search enormously. Then add context. Scheduler signals like KV-cache utilization and how often requests get preempted explain why the queue grew. Per-GPU signals like memory bandwidth use catch a single slow or stuck instance. The trap is reaching for cluster-wide throughput. A daily average tokens per second number is far too coarse. It stays flat while the worst requests blow up, so it can never localize a tail-latency problem.

**P99 TTFT** decomposes cleanly into two per-request components. The first is queue wait, time in the admission queue before scheduling. The second is prefill, the forward pass over the prompt. You need a labeled span for each. Per-request queue wait and per-request prefill duration let you isolate which dimension regressed. That is the first cut, and it is essential. Next you need the why. Scheduler-level state explains a growing queue. KV-cache utilization fraction and preemption rate are the usual culprits. When the cache fills, the scheduler preempts in-flight requests, and queue wait spikes for the tail. Per-GPU resource metrics form the tiebreaker. HBM bandwidth use and tensor-core occupancy catch a single oversubscribed or wedged replica that serves slowly without throwing errors. Cluster P50 throughput is the wrong tool. It is an aggregate of the wrong statistic. A P50 stays flat while the P99 tail explodes, and tokens per second never attributes a latency regression to a request. In practice you capture queue wait, prefill, scheduler state, and GPU counters as OpenTelemetry spans on every request. That gives you the full decomposition to localize the regression fast.