Time per Output Token

Per-token decode time grows linearly as a chat session lengthens, debug it

TPOT is high, so the team plans to upgrade A100 to a faster compute GPU, critique

Find the wrong move in 'decode is slow, so let's switch to a smaller-FLOP model'

TTFT: name the metric it captures and what dominates it

In benchmarks, tokens per second measures what exactly?

End-to-end latency: fill in the queue, prefill and decode components.

Match TTFT and TPOT to the latency they measure during streaming

Define TTFT, TPOT and throughput. Which should a chatbot vs a batch summarizer optimize?

A chatbot team reports TTFT P99 jumped from 300ms to 1.4s overnight. Which root cause is most likely?

What problem do DistServe and Splitwise solve by separating prefill and decode onto different GPUs?

For a 200-token prompt and 1000-token completion, why does decode dominate end-to-end latency?

How does Sarathi-Serve's chunked prefill change the latency profile of a serving system?

Predict the throughput and per-request latency behaviour as batch size grows past saturation