Why production chat UIs stream tokens token by token instead of returning the full reply

Flashcard·Easy·4.0 · 0·~30s·Asked atFreshworks Mongodb Weaviate·Relevant atOpenAI

Attempt it

Streaming collapses perceived latency from TTFT + N * TPOT (wait for the whole reply) down to TTFT alone (wait for the first token). The user starts reading as soon as decoding begins, and their reading speed roughly overlaps the model's decoding speed for the rest of the response, so the WAIT, not the total work, is what shrinks. For a 1000-token reply at 50 tok/s, that turns ~20 seconds of blank screen into a few hundred ms before the first word appears. The cost is small (an open HTTP connection and per-chunk SSE framing) and the UX win is order of magnitude.

Your answer

**Streaming does not make the server faster. It makes the wait shorter.** The total time the GPU spends decoding is identical whether the response is streamed or buffered. **The decomposition.** Latency for a 1000-token reply at 50 tokens per second has two parts. Time to first token (TTFT) might be 300 milliseconds. Then decode time is 1000 / 50 = 20 seconds. **Buffered response.** The user waits 300 ms plus 20 seconds = 20.3 seconds staring at a blank screen, then sees the entire reply at once. **Streamed response.** The user waits 300 ms (TTFT), sees the first word, and starts reading. The model continues decoding at 50 tokens per second. The user's reading speed is roughly 4-5 words per second, which is similar to the model's tokens per second rate. The two streams overlap. **The perceived wait** with streaming is just the 300 ms TTFT. The rest of the decode happens while the user is reading. **The cost.** Streaming requires keeping the HTTP connection open for the duration of the response, and sending per-chunk Server-Sent Events (SSE) framing. The server-side cost is small. The UX win is order of magnitude on long responses. This is why every production chat UI (ChatGPT, Claude, Gemini) streams by default.

**Streaming collapses perceived latency.** The model does the same amount of work either way, but the user's experience of waiting is completely different. **The latency formula.** Total response latency decomposes as `TTFT + N * TPOT`, where TTFT is the time to first token (includes prefill and queue wait), N is the number of output tokens, and TPOT is the per output token time. For a 1000-token response at 50 tokens per second, the formula gives 0.3 + 1000 * 0.02 = 20.3 seconds. **Buffered serving** forces the user to wait for the full 20.3 seconds before seeing anything. The screen stays blank for 20 seconds of decode, then renders the full reply at once. This feels broken to users for any response longer than a few sentences. **Streaming serving** sends each decoded token (or small chunk of tokens) to the client as soon as it is produced. The user sees the first word after TTFT (0.3 seconds), then sees subsequent words as they arrive at 50 per second. **The overlap that makes it work.** Adult reading speed is roughly 200-250 words per minute, or 3-4 words per second. Model decode speed for production-class LLMs runs 40-80 tokens per second per stream, which is roughly 4-8 words per second. The two rates are comparable, so the user finishes reading earlier portions of the response while the model is still decoding later portions. The decode time is hidden behind the user's reading time. **The perceived wait** drops from 20.3 seconds (TTFT + full decode) to 0.3 seconds (TTFT alone). The actual server work is unchanged. The UX win is dramatic. **Implementation cost.** Streaming requires Server-Sent Events (SSE) or WebSocket framing, an open HTTP connection for the response duration, and per-chunk serialization. None of these are expensive. The total operational overhead is a few percent of total request cost.

TL;DR

Streaming hides decode time behind the user's reading speed. The wait drops from TTFT plus full decode to TTFT alone, turning 20 seconds of blank screen into a few hundred milliseconds.

Memory aid

Easy to grasp

Imagine ordering at a restaurant where the kitchen could either hand you the full meal in 30 minutes or bring out each dish as it finishes. The total cook time is the same either way, but the wait feels completely different. With streaming dishes, you start eating in five minutes and the kitchen finishes the rest while you enjoy the appetizer. With the all at once approach, you stare at an empty table for half an hour. LLMs work the same way. The server takes the same time to decode either way, but streaming lets the user start reading the response immediately, so the wait shrinks to just the time before the first word appears.

Key concepts

Streaming responses Time to first token Perceived latency

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Streaming is the defining UX pattern of modern LLM chat applications, and it works by exploiting a temporal overlap between model decoding and user reading. The model does the same amount of work whether or not the response is streamed; what changes is when the user starts seeing tokens, and that single change collapses the perceived wait from tens of seconds to a few hundred milliseconds.

Understanding streaming is foundational because it shifts the right SLA metric from wall-clock response latency to time to first token, which has cascading effects on how production serving is designed. Prefill-decode disaggregation, queue management, autoscaling signals, and tail latency optimization all aim at TTFT in streaming-first deployments.

This deep dive walks through the latency math, the user reading speed overlap that makes streaming work, the implementation specifics of Server-Sent Events, the workloads where streaming has no benefit, and the operational considerations that come with long-lived connections. By the end you should be able to defend why every production chat UI streams, identify which workloads do not benefit, and reason about the right latency SLOs for streamed deployments.

The latency formula and where streaming acts

Total response latency for an LLM request decomposes as:

\text{Total} = \text{TTFT} + N \cdot \text{TPOT}

Where TTFT is the time to first token (queue wait plus prefill plus first decode step), N is the number of output tokens, and TPOT is the per output token decode time at steady state.

Typical values for a production deployment: TTFT 200-500 ms (depending on prompt length and queue load), TPOT 15-30 ms per token (depending on hardware and quantization).

For a 1000-token reply, this gives 0.3 + 1000 * 0.02 = 20.3 seconds wall-clock. For a buffered response, this is also the perceived wait: the user sees nothing until decode completes.

Streaming pushes each token to the client as it is produced. The first token arrives after TTFT. Subsequent tokens arrive every TPOT. The user is no longer waiting for the entire response; they are watching it unfold.

Why streaming works: the reading-speed overlap

Implementation: Server-Sent Events and chunked responses

Operational considerations of long-lived connections

Where streaming does not help

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

ChatGPT, Claude, and Gemini all stream by default; the streaming UX defined the category.
OpenAI's GPT-5.5 API ships an explicit 'stream: true' parameter and is the most-used path for chat applications.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does TTFT decompose further into prefill versus queue components?

TTFT = queue wait + prefill compute + first decode step. Queue wait is set by load and admission control. Prefill compute scales with prompt length and hardware (~50-200 ms for typical prompts on H100). First decode step is one TPOT (~20 ms). Optimizing TTFT means reducing all three; prefill is the largest typically and is the target of prefill-decode disaggregation.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Thinking streaming makes the model faster. It does not change decode time at all; it overlaps decode with the user's reading time, collapsing the perceived wait.

60 second bullets to scan on the way to the call.

Decompose total response latency into TTFT and N times TPOT
Identify which term streaming hides