Same topic, related formats. Practice these next.
Define TTFT, TPOT, and aggregate throughput for an LLM serving system. Then describe which metric a real-time chatbot should prioritize versus an offline batch document-summarization job. Justify each with the user-visible behavior.
TTFT is time to first token, TPOT is time per output token after that, throughput is cluster tokens per second. Chat optimizes the first two, batch jobs optimize the third.
Imagine a restaurant kitchen. TTFT is how long after you order before the first dish lands on your table. TPOT is how fast each later dish arrives once food starts coming. Throughput is how many total dishes the whole kitchen serves per hour across every table. A diner who is hungry and watching cares about the first two, you want food soon and a steady flow. A catering company filling a thousand boxes overnight only cares about the total count by morning, nobody is sitting and waiting. So a chatbot tunes for the waiting diner, and a batch summarizer tunes for the catering count. Same kitchen, same stoves, but you turn different knobs depending on who is watching the clock.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: define all three metrics, map each to a phase (prefill, decode, fleet), explain the batch-size tradeoff, give chat targets, name chunked prefill and disaggregation, and defend a tail percentile.
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating latency and throughput as the same knob. They oppose each other. Bigger batches lift throughput but raise per-request latency, so you cannot maximize both at once.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.