A chatbot team reports TTFT P99 jumped from 300ms to 1.4s overnight. Which root cause is most likely?

Attempt it

Quick reference

TL;DR

TTFT, TPOT, and throughput pull against each other; a sudden TTFT spike points at queue wait or prefill, not at per-token decode speed or matmul throughput.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of a busy coffee shop. TTFT is how long until the barista starts making your drink. TPOT is how fast each sip-sized step of your drink gets made. Throughput is how many drinks the whole shop finishes per hour. If you batch more customers together, the shop finishes more drinks per hour, but any one person waits longer to even get started and feels slower sip to sip. So a sudden jump in how long until your drink is started usually means the line got longer or your order got bigger, not that the barista's hands slowed down. You pick which number to protect based on whether customers care most about waiting, sipping, or total shop output.

Key concepts

Time to first token Time per output token Throughput Continuous batching

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define TTFT, TPOT, throughput + the batching tradeoff + decompose TTFT into queue plus prefill + why the distractors confuse metrics + which metric a product should protect.

Real-world examples

Real products, models, and research that use this idea.

vLLM exposes TTFT and inter-token latency as separate Prometheus metrics so teams can set an SLO on each one independently rather than one blended latency.
NVIDIA TensorRT-LLM uses chunked prefill so a long incoming prompt does not block ongoing decode streams, protecting TPOT while keeping throughput high.
Anthropic and OpenAI ship streaming chat APIs where TTFT is the user-felt responsiveness metric for Claude Opus 4.7 and GPT-5.5, while their batch APIs optimize pure throughput at relaxed TTFT.
SGLang's continuous batching scheduler admits requests at the token level so queue wait stays low even as concurrency rises, holding P99 TTFT down.
Together AI and Fireworks expose per-endpoint TTFT and tokens-per-second dashboards so customers pick a serving tier by the metric their product needs.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does increasing batch size raise throughput but worsen TTFT and TPOT for individual requests?

Sign in or go Pro to see the approach for this follow-up.

QHow does chunked prefill change the TTFT versus TPOT tradeoff?

Sign in or go Pro to see the approach for this follow-up.

QIf P99 TTFT regressed but P50 stayed flat, what does that tell you?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating TTFT, TPOT, and throughput as one latency number. They trade off against each other, so a regression in one says nothing reliable about the others without a trace.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.