What traces and metrics do you need to debug a P99 TTFT regression in production LLM serving?
A team's P99 TTFT regressed from 400ms to 1.8s overnight while the model and code are unchanged. Describe the observability you'd want and how you'd use it to localize the regression to a root cause. List the specific metrics and traces involved.
Split TTFT into queue_wait plus prefill, instrument each as a per-request span, then read scheduler state for queue blowups and per-GPU counters plus prompt-length drift for prefill blowups.
Imagine a restaurant where the wait for your first bite is the number you care about. That wait has two parts: how long you stood in line before a table opened, and how long the kitchen took once you sat down. If first-bite time doubled overnight, you must know which part grew. Maybe the line got long because the dining room is full and nobody leaves. Maybe cooking got slow because one stove broke, or because everyone suddenly ordered huge meals. A serving system is the same. You stamp every order with a clock at each step, so you can see whether the line or the kitchen caused the slowdown. Without those stamps you are only guessing which half of the wait went wrong.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Time to first token is the latency that decides whether a streaming LLM product feels responsive. It is the wall-clock gap between a request arriving and the first output token appearing. When P99 TTFT regresses from 400ms to 1.8s overnight with the model and code unchanged, the regression is real, it lives in the serving system or the workload, and the only way to find it fast is to have instrumented the system so the number can be taken apart.
The central skill this question tests is decomposition under pressure. A senior engineer does not stare at a TTFT dashboard and start swapping GPUs. They split TTFT into its two additive components, queue wait and prefill duration, read the P99 of each, and let the bigger mover point at the right family of metrics. Each branch has a small, well-defined set of signals that confirm or kill a hypothesis.
This deep dive walks through the decomposition, the per-request spans you stamp, the scheduler metrics that explain queue-wait regressions, the per-GPU and workload metrics that explain prefill regressions, and the correlation discipline that ties request traces to infrastructure counters. By the end you should be able to narrate a confident root-cause path from a P99 alert to a one-line answer.
Decompose TTFT before touching anything
TTFT is not atomic. It is the sum of the time a request spends waiting to be admitted onto the GPU plus the time the prefill forward pass takes to produce token one.
Queue wait is the interval from request arrival to the first scheduled step. Prefill duration is the interval from admission to the emission of the first token. These map to two distinct subsystems: the scheduler and admission control on one side, the GPU compute path on the other.
The operational reason to split first is falsifiability. If you only have total TTFT, every theory is consistent with the data, so you end up swapping hardware or restarting nodes on a hunch. Once you have the P99 of each span side by side against a baseline, the regression announces which subsystem owns it. A 1.4s jump that lands entirely in queue wait is a scheduler story. The same jump in prefill is a compute or workload story. You almost never need to investigate both branches at once.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM exposes Prometheus metrics for KV cache usage, running and waiting request counts, and preemptions, the exact gauges you read to localize a queue-wait regression.
- NVIDIA TensorRT-LLM with the Triton backend emits per-request and per-batch latency stats plus GPU utilization counters for prefill attribution.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you distinguish a degraded GPU from a noisy-neighbor process on a shared host?
Both raise prefill duration on one card. A degraded GPU shows persistently low tensor-core occupancy plus thermal throttle or ECC errors in DCGM. A noisy neighbor shows the device busy with work your request did not issue, so SM occupancy is high but your kernel waits. Correlate device counters with your own per-request kernel timeline.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating TTFT as one opaque number. Without splitting queue wait from prefill, you cannot tell a scheduler admission problem from a slow prefill, so every fix is a guess.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.