Why production chat UIs stream tokens token by token instead of returning the full reply

Attempt it

Streaming collapses perceived latency from TTFT + N * TPOT (wait for the whole reply) down to TTFT alone (wait for the first token). The user starts reading as soon as decoding begins, and their reading speed roughly overlaps the model's decoding speed for the rest of the response, so the WAIT, not the total work, is what shrinks. For a 1000-token reply at 50 tok/s, that turns ~20 seconds of blank screen into a few hundred ms before the first word appears. The cost is small (an open HTTP connection and per-chunk SSE framing) and the UX win is order of magnitude.

Your answer

**Streaming does not make the server faster. It makes the wait shorter.** The total time the GPU spends decoding is identical whether the response is streamed or buffered. **The decomposition.** Latency for a 1000-token reply at 50 tokens per second has two parts. Time to first token (TTFT) might be 300 milliseconds. Then decode time is 1000 / 50 = 20 seconds. **Buffered response.** The user waits 300 ms plus 20 seconds = 20.3 seconds staring at a blank screen, then sees the entire reply at once. **Streamed response.** The user waits 300 ms (TTFT), sees the first word, and starts reading. The model continues decoding at 50 tokens per second. The user's reading speed is roughly 4-5 words per second, which is similar to the model's tokens per second rate. The two streams overlap. **The perceived wait** with streaming is just the 300 ms TTFT. The rest of the decode happens while the user is reading. **The cost.** Streaming requires keeping the HTTP connection open for the duration of the response, and sending per-chunk Server-Sent Events (SSE) framing. The server-side cost is small. The UX win is order of magnitude on long responses. This is why every production chat UI (ChatGPT, Claude, Gemini) streams by default.

**Streaming collapses perceived latency.** The model does the same amount of work either way, but the user's experience of waiting is completely different. **The latency formula.** Total response latency decomposes as `TTFT + N * TPOT`, where TTFT is the time to first token (includes prefill and queue wait), N is the number of output tokens, and TPOT is the per output token time. For a 1000-token response at 50 tokens per second, the formula gives 0.3 + 1000 * 0.02 = 20.3 seconds. **Buffered serving** forces the user to wait for the full 20.3 seconds before seeing anything. The screen stays blank for 20 seconds of decode, then renders the full reply at once. This feels broken to users for any response longer than a few sentences. **Streaming serving** sends each decoded token (or small chunk of tokens) to the client as soon as it is produced. The user sees the first word after TTFT (0.3 seconds), then sees subsequent words as they arrive at 50 per second. **The overlap that makes it work.** Adult reading speed is roughly 200-250 words per minute, or 3-4 words per second. Model decode speed for production-class LLMs runs 40-80 tokens per second per stream, which is roughly 4-8 words per second. The two rates are comparable, so the user finishes reading earlier portions of the response while the model is still decoding later portions. The decode time is hidden behind the user's reading time. **The perceived wait** drops from 20.3 seconds (TTFT + full decode) to 0.3 seconds (TTFT alone). The actual server work is unchanged. The UX win is dramatic. **Implementation cost.** Streaming requires Server-Sent Events (SSE) or WebSocket framing, an open HTTP connection for the response duration, and per-chunk serialization. None of these are expensive. The total operational overhead is a few percent of total request cost.

Quick reference

TL;DR

Streaming hides decode time behind the user's reading speed. The wait drops from TTFT plus full decode to TTFT alone, turning 20 seconds of blank screen into a few hundred milliseconds.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine ordering at a restaurant where the kitchen could either hand you the full meal in 30 minutes or bring out each dish as it finishes. The total cook time is the same either way, but the wait feels completely different. With streaming dishes, you start eating in five minutes and the kitchen finishes the rest while you enjoy the appetizer. With the all at once approach, you stare at an empty table for half an hour. LLMs work the same way. The server takes the same time to decode either way, but streaming lets the user start reading the response immediately, so the wait shrinks to just the time before the first word appears.

Key concepts

Streaming responses Time to first token Perceived latency

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: decompose latency into TTFT and TPOT, explain the user reading speed overlap with model decode speed, compute the perceived wait with and without streaming, and close with operational tradeoffs (connection management, structured output limits).

Real-world examples

Real products, models, and research that use this idea.

ChatGPT, Claude, and Gemini all stream by default; the streaming UX defined the category.
OpenAI's GPT-5.5 API ships an explicit 'stream: true' parameter and is the most-used path for chat applications.
Anthropic's Claude Opus 4.7 API streams tokens via SSE for both web and API consumers.
Llama 4 Maverick endpoints at Together AI and Fireworks stream by default; non-streaming is the opt-in.
GitHub Copilot streams completions character by character to overlap with developer reading and review.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does TTFT decompose further into prefill versus queue components?

Sign in or go Pro to see the approach for this follow-up.

QWhy is TPOT less user-perceived than TTFT once streaming is on?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes when LLM output is faster than human reading speed?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking streaming makes the model faster. It does not change decode time at all; it overlaps decode with the user's reading time, collapsing the perceived wait.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.