A chatbot team reports TTFT P99 jumped from 300ms to 1.4s overnight. Which root cause is most likely?
TTFT, TPOT, and throughput pull against each other; a sudden TTFT spike points at queue wait or prefill, not at per-token decode speed or matmul throughput.
Think of a busy coffee shop. TTFT is how long until the barista starts making your drink. TPOT is how fast each sip-sized step of your drink gets made. Throughput is how many drinks the whole shop finishes per hour. If you batch more customers together, the shop finishes more drinks per hour, but any one person waits longer to even get started and feels slower sip to sip. So a sudden jump in how long until your drink is started usually means the line got longer or your order got bigger, not that the barista's hands slowed down. You pick which number to protect based on whether customers care most about waiting, sipping, or total shop output.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
TTFT, TPOT, and throughput are the three numbers that define LLM serving performance, and the single biggest sign of a junior answer is collapsing them into one word: 'latency'. They measure different things, they are felt by different people, and crucially they trade off against one another. You cannot maximize all three at once; you pick the one your product lives or dies on and tune the serving stack around it.
This question is a debugging scenario, but the skill it tests is the mental model behind it. A team sees P99 time-to-first-token leap from 300ms to 1.4s overnight. The right answer requires knowing exactly what TTFT is made of, what it is NOT sensitive to, and why three plausible-sounding distractors each point at the wrong metric. Each distractor is engineered to be tempting: it names a real serving concept and a real failure mode, but attaches it to the wrong metric. Untangling that mapping is exactly the competence an interviewer is probing.
This deep dive defines the three metrics, explains the batching tension that couples them, decomposes TTFT into its two real components, and then walks the four options to show why queue depth or prompt length is the only root cause consistent with a TTFT-only regression. It closes with the question every serving team eventually faces: given that you cannot win all three, which one should you protect? By the end you should be able to read a serving trace and say which knob moved which metric, and defend why a chat product and a batch pipeline make opposite choices.
The three metrics and who feels them
Time to first token (TTFT) is the delay from request arrival until the first output token reaches the user. It is the responsiveness a chat user feels: the pause before anything appears on screen.
Time per output token (TPOT), sometimes called inter-token latency, is the steady-state interval between subsequent tokens during streaming. It sets how fast text scrolls once it starts. A user reads at a fixed pace, so once TPOT drops below roughly the reading speed, further gains are invisible.
Throughput is a system-level number, not a per-user one: total tokens per second the GPU produces across all concurrent requests. It is what the finance team cares about, because it sets cost per token. A serving operator optimizing for cost will push throughput; a product team optimizing for feel will push TTFT. These are different masters, and that is the heart of the tension.
Notice the asymmetry of perception. TTFT and TPOT are per-request and human-felt: a single user can tell you their request was slow. Throughput is aggregate and invisible to any one user, but it determines the bill. Because the people who feel each metric are different, teams that report only one blended 'latency' number routinely ship the wrong optimization. They tune the GPU for tokens per second, watch the dashboard look healthy, and then field complaints that the chat feels laggy, because the metric users actually feel, first-token time, was never on the dashboard at all.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM exposes TTFT and inter-token latency as separate Prometheus metrics so teams can set an SLO on each one independently rather than one blended latency.
- NVIDIA TensorRT-LLM uses chunked prefill so a long incoming prompt does not block ongoing decode streams, protecting TPOT while keeping throughput high.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does increasing batch size raise throughput but worsen TTFT and TPOT for individual requests?
Decode is memory-bandwidth-bound: reading weights from HBM dominates. A batch amortizes that one read across many requests, lifting tokens per second. But each request now waits for a batch slot and shares every decode step, so its own TTFT and TPOT grow.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating TTFT, TPOT, and throughput as one latency number. They trade off against each other, so a regression in one says nothing reliable about the others without a trace.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.