Walk through how to chart tokens per second separately for prefill and decode
You are debugging a self-hosted model whose total latency is fine on average but tail latency is bad. You want to chart tokens per second separately for prefill (input processing) and decode (output generation). Walk through the instrumentation and the chart shape.
Record three timestamps per generation; compute prefill_tps and decode_tps; chart both as percentile bands grouped by model and input-length bucket.
Think of running a race in two halves. The first half is sprinting to the starting line and getting your shoes tied (prefill). The second half is the actual race (decode). If you only track the total time, you cannot tell whether a slow finish was because shoe-tying took forever or because the runner was slow on the track. Split the timer in two: one for shoe-tying, one for running. Now when your overall time gets worse, you know which half to blame. For LLMs, the time to first token is your shoe-tying line; everything before it is prefill, everything after it is decode.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Total LLM latency is two phases stacked: prefill processes the input tokens to populate the KV cache, and decode generates output tokens one at a time. The two phases have different bottlenecks, scale differently with input, and degrade for different reasons. Charting total latency as a single number is a category error that costs hours of debugging when tail latency goes bad in production.
This walkthrough is the instrumentation and dashboard pattern that production self-hosted LLM teams converged on. It splits each generation call into two throughput numbers, records both as span attributes, charts them as percentile bands grouped by model and input-length bucket, and gives you a root-cause map for translating regressions in either phase into actionable serving-stack changes.
Mental model: every LLM call is a two-stage pipeline. The time to first token splits the stages. Throughput in each stage has its own bottleneck and its own fix.
Prefill vs decode: what is actually happening
Prefill
Prefill processes all input tokens in parallel through the model's transformer layers, writing K and V vectors for each token into the KV cache. The operation is fundamentally a large matrix multiplication, compute-bound, and proportional to input length. A 4K-token prefill takes about four times the wall-clock of a 1K-token prefill on a saturated GPU.
Decode
Decode generates output tokens one at a time. Each token reads the entire KV cache to compute attention, then produces one new K, V, and output token. Decode is memory-bound, not compute-bound: the GPU spends most of its time pulling the cache through HBM bandwidth. Output token N takes roughly the same time as token N-1; latency per token is roughly constant across the output stream.
Why total latency hides both
A call with a 4K input and 200-token output spends most of its time in prefill. A call with a 200-token input and 4K-token output spends most of its time in decode. Total latency averaged across both kinds of calls obscures whichever phase is regressing.
Time to first token as the boundary
The split is observable from the client. Prefill ends when the first output token is ready to stream; decode begins immediately after. So t_first_token - t0 is essentially prefill latency (plus queue time), and t_end - t_first_token is decode latency. This is why every modern LLM serving framework treats time to first token (ttft) as a first-class metric.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM 0.6+ exposes prefill and decode metrics natively via its Prometheus endpoint; standard pattern for self-hosted Llama 4 Maverick or Qwen 3.5 deployments.
- TGI (Hugging Face Text Generation Inference) reports time to first token and inter token latency as separate Prometheus series.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you alert on prefill regression specifically without false-flagging decode regressions?
Two independent rolling-baseline alerts, one per metric. Prefill alert keys off p99 prefill_tps in the 2K+ input-length bucket; decode alert keys off p99 decode_tps. Each routes to a different runbook (prefill -> queue/scheduler runbook, decode -> KV-cache/batch runbook).
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Charting only mean throughput. p50 hides the tail problems that are the actual user complaint; percentile bands per phase are mandatory.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.