Which metrics are essential to debug a P99 TTFT regression in production LLM serving?
Debug a P99 TTFT regression with per-request queue wait and prefill spans, scheduler state like KV-cache pressure and preemptions, and per-GPU resource use; cluster P50 throughput is too coarse.
Imagine a busy coffee shop. A customer's wait from ordering to first sip has two parts: time standing in line, and time the barista spends making it. To fix slow service you must measure both parts separately, and for the unlucky customers, not just the average. You also watch the shop floor: is the line backing up because the counter is full, or because one barista is stuck? A single number like drinks per day tells you nothing, because it stays flat even when the slowest customers wait ages. Good debugging tracks each customer's journey and the shop's live state together. That is what observability for model serving does: it follows each request while watching the system at the same time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Time to first token is the latency a user feels before any output appears, and at the tail it is the metric that decides whether a product feels responsive or broken. A P99 TTFT regression means the slowest one percent of requests degraded. The median user may notice nothing, which is exactly why aggregate dashboards lie to you here.
The core skill this question tests is observability discipline under a real incident. You are not asked to list every metric a serving stack emits. You are asked which signals genuinely let you localize and attribute a tail-latency regression, and which are vanity numbers that feel relevant but cannot move the investigation forward.
The answer rests on one structural fact: TTFT is a sum of measurable phases, and a regression must live in one of them. Decompose the metric into per-request components, attribute the bad component to a system cause, and confirm with resource counters. The metric that fails this test, cluster P50 throughput, fails for a precise reason worth being able to articulate.
Decomposing TTFT into per-request phases
Every request that reaches a serving replica passes through two phases before the first token appears. It waits in an admission queue, then it runs prefill, the parallel forward pass that reads the entire prompt and produces the first output token. TTFT is essentially their sum.
The operational consequence is that you must measure both as separate per-request spans, not as one blended number. If you only record total TTFT you can detect that the tail moved, but you cannot say which phase moved it. The two phases have entirely disjoint failure modes. Queue wait is governed by the scheduler and the memory budget, while prefill is governed by prompt length and the health of the compute path. Blending them into one number throws away exactly the signal that separates a capacity problem from a compute problem.
With both spans labeled you split the regression on the very first query. You look at the P99 of each component over the regression window and read off which one moved. If queue wait went from 50 ms to 1.5 s while prefill held flat, the model forward pass is innocent and the bottleneck is admission. If prefill doubled while queue wait stayed flat, the queue is fine and the compute path slowed down. One graph, one decision, no guessing.
This is why options A and B in the question are both essential rather than redundant. They are not two views of the same thing. They are the two independent terms of the sum, and a regression in either is invisible in the other. Dropping either one leaves a blind spot that a real incident will eventually fall into.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM exposes Prometheus metrics for queue time, KV-cache utilization, and preemption counts that map directly onto this TTFT decomposition.
- OpenTelemetry GenAI semantic conventions standardize per-request spans for TTFT and inter-token latency across serving stacks in 2026.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does KV-cache utilization predict queue-wait spikes in a continuous-batching scheduler?
Trace the admission loop. The scheduler packs requests until the cache is full. A long-context arrival forces preemption and recompute of victims, which inflates their queue wait. Plot preemption rate against P99 queue wait and the correlation is direct.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Watching cluster P50 throughput to chase a P99 latency bug. Averages stay flat while tails blow up, and aggregate throughput hides the unlucky requests you actually need to attribute.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.