A chatbot team reports TTFT P99 jumped from 300ms to 1.4s overnight. Which root cause is most likely?

LLM serving has three latency numbers, and they are not the same thing. **Time to first token** (TTFT) is how long the user waits before the very first word appears. Time per output token (TPOT) is how fast each subsequent word streams out. Throughput is how many tokens per second the whole GPU produces across all users at once. These three pull against each other. Putting more requests into one batch raises throughput, because the GPU reads the weights once and serves many users from that read. But each individual user now shares the GPU, so their TTFT and TPOT get worse. You cannot maximize all three at the same time. In the question, TTFT jumped from 300ms to 1.4s overnight. TTFT is built from two pieces: how long the request waited in the admission queue, plus how long prefill took to process the prompt. A jump that large means either more users showed up (queue grew) or prompts got longer (prefill grew). The wrong answers blame per-token decode speed or matmul throughput. Those affect TPOT and tokens per second, not first-token time. The model also has no 'thinking time' before token one; it emits at a fixed rate once prefill finishes.

**TTFT, TPOT, and throughput** are three distinct serving metrics that trade off against each other. TTFT is the wait before the first token. TPOT is the streaming interval between later tokens. Throughput is total tokens per second across all concurrent requests. The tension comes from batching. Larger batches amortize the cost of reading model weights from memory, so throughput climbs. But each request shares the GPU, so its TTFT and TPOT both degrade. You tune the batch size and scheduler for whichever metric your product needs. TTFT decomposes into queue wait plus prefill time. So a sudden 4-5x TTFT regression almost always means one of two things. Either concurrency rose and requests now sit longer before admission. Or prompts grew longer, so prefill takes more compute. Both are visible in a serving trace. The distractors fail on this decomposition. A GQA group-size change moves per-step decode cost, which is TPOT, not first-token time. A matmul kernel regression hurts steady-state throughput, but TTFT is mostly set before that matters. The model does not deliberate before the first token. It emits at a fixed rate once prefill completes, so 'thinking longer' is not a real mechanism here.