Which metrics are essential to debug a P99 TTFT regression in production LLM serving?

Attempt it

Quick reference

TL;DR

Debug a P99 TTFT regression with per-request queue wait and prefill spans, scheduler state like KV-cache pressure and preemptions, and per-GPU resource use; cluster P50 throughput is too coarse.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a busy coffee shop. A customer's wait from ordering to first sip has two parts: time standing in line, and time the barista spends making it. To fix slow service you must measure both parts separately, and for the unlucky customers, not just the average. You also watch the shop floor: is the line backing up because the counter is full, or because one barista is stuck? A single number like drinks per day tells you nothing, because it stays flat even when the slowest customers wait ages. Good debugging tracks each customer's journey and the shop's live state together. That is what observability for model serving does: it follows each request while watching the system at the same time.

Key concepts

Time to first token Distributed tracing Tail latency KV cache

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: decompose TTFT into queue wait + prefill, attribute queue growth via KV-cache utilization and preemptions, use per-GPU counters as tiebreaker, explain why P50 throughput is the wrong statistic for a P99 regression.

Real-world examples

Real products, models, and research that use this idea.

vLLM exposes Prometheus metrics for queue time, KV-cache utilization, and preemption counts that map directly onto this TTFT decomposition.
OpenTelemetry GenAI semantic conventions standardize per-request spans for TTFT and inter-token latency across serving stacks in 2026.
NVIDIA Triton and TensorRT-LLM emit per-GPU HBM bandwidth and tensor-core occupancy via DCGM, used to spot a single wedged replica.
Grafana dashboards over vLLM and SGLang plot P99 TTFT alongside scheduler preemption rate to correlate tail spikes with cache pressure.
Anthropic and OpenAI production serving teams alert on per-percentile TTFT, not aggregate throughput, because tail latency drives user-perceived quality.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does KV-cache utilization predict queue-wait spikes in a continuous-batching scheduler?

Sign in or go Pro to see the approach for this follow-up.

QHow would you separate a prefill regression caused by lost prefix caching from one caused by longer prompts?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the danger of alerting on P99 TTFT alone without per-request span attribution?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Watching cluster P50 throughput to chase a P99 latency bug. Averages stay flat while tails blow up, and aggregate throughput hides the unlucky requests you actually need to attribute.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.