Define TTFT, TPOT and throughput. Which should a chatbot vs a batch summarizer optimize?

Three numbers describe how an LLM server feels to use. **TTFT**, time to first token, is the wait from when your request arrives until the first word streams back. It is set mostly by queue waiting plus prefill, the pass that reads your whole prompt. TPOT, time per output token, is the gap between each streamed word after the first. It is set by the decode phase, where the model emits one token at a time. A low TPOT makes the answer feel like it is being typed quickly and smoothly. Throughput is a system number, not a per-user one. It counts the total tokens per second the whole cluster produces across every request at once. You raise it by batching many requests together so the GPU stays busy. Here is the tension. Big batches share the GPU's fixed memory reads across many users, lifting throughput. But each user now waits behind more work, so their TTFT and TPOT get worse. A chatbot keeps batches small to feel snappy. A batch summarizer packs batches full because no human is watching the clock.

**TTFT** is the wall-clock time from request arrival to the first streamed token. It is dominated by queue wait plus prefill, the parallel pass over the prompt. Long prompts and busy queues both inflate it. TPOT is the average time between successive tokens during streaming. It reflects decode-step latency, which is memory-bandwidth-bound and scales with batch size. Throughput is the aggregate tokens per second across the whole cluster, maximized when batches run near saturation. The core tension is that latency and throughput pull apart. Large batches amortize fixed memory reads, so throughput climbs. But each request waits behind more work, so TTFT and TPOT rise. A real-time chatbot optimizes TTFT and TPOT. Users judge responsiveness by when text appears and how fluid the stream feels. A common target is TTFT under 500 ms and TPOT under 30 ms, which outpaces reading speed. You cap batch size below saturation to protect those numbers. An offline batch summarizer optimizes throughput per dollar. No human watches a stream, so TTFT and TPOT are invisible. You run at maximum sustainable batch to squeeze every token per second from each GPU.