Total decode time is identical with or without streaming. Why does Server-Sent Events (SSE) streaming dramatically improve the user perceived latency of an LLM application?
Streaming pushes each token as it is generated, so the user sees content at time-to-first-token instead of waiting for the whole reply. Total decode time is unchanged; only the perceived wait shrinks.
Imagine asking a friend a long question. One friend stays silent, works out the entire answer in their head, and recites all of it at once after a long pause. Another friend starts talking the moment the first words come to mind and keeps going. Both take the same total time to finish, but the second feels far more responsive, because you stop waiting almost immediately. Streaming makes the model behave like that second friend. The server sends each word the instant it is ready instead of holding everything back. You start reading right away, and the rest flows in as it is produced. The actual work the model does is identical either way.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Token streaming is the most visible UX decision in any LLM application, and also the one most often misunderstood in interviews. The trap is simple. Streaming feels faster, so candidates describe it as a performance optimization. It is not. It changes nothing about how fast the model generates tokens. It changes only when the user first sees content. Getting that one distinction right, and being able to defend it with numbers, is what separates a confident answer from a hand-wave.
An LLM decodes autoregressively, emitting one token at a time. A 1000-token reply at 50 tokens per second takes roughly 20 seconds, and that number is fixed by the decode loop regardless of how the bytes reach the browser. Streaming does not touch throughput, tokens per second, or total decode time. What it touches is perceived latency: the gap between the user pressing send and seeing the first scrap of a response. That gap is where every chat product lives or dies, because a blank screen reads as a broken product while flowing text reads as a fast one.
This deep dive separates actual latency from perceived latency, shows why time-to-first-token is the metric that governs felt responsiveness, explains why Server-Sent Events is the standard transport and how it compares with WebSockets and raw chunked transfer, and works through the modest costs of holding a connection open. By the end you should be able to defend the claim that streaming is a pure UX win, quantify it, and connect it to what you would actually optimize to make an app feel fast.
Actual latency versus perceived latency
Actual latency is the total wall-clock time to produce the complete response. It is set by two phases. Prefill processes the prompt and produces the first token. Decode then generates the remaining tokens one at a time, bounded by the per-token decode rate. The sum of those two phases is fixed by the model, the hardware, and the request shape, and no delivery trick can shorten it.
Perceived latency is different. It is the time between the user action and the first visible content. Humans tolerate a response that is still arriving far better than a blank screen, because partial output is a continuous signal that the system is working. Interface research has long held that delays under a second feel instantaneous, while delays past a few seconds break the sense of direct interaction. A 20-second blank wait sits deep in the territory where users assume failure and reach for the refresh button.
Streaming exploits exactly this asymmetry. With buffering, perceived latency equals actual latency, because nothing appears until the last token lands. With streaming, perceived latency collapses to time-to-first-token while actual latency stays put. For a 20-second reply with a 300-millisecond first token, that is roughly a 60x reduction in the wait that the user actually feels. The model did the same work in the same time. Only the moment of first feedback moved, and that single move is what converts an unusable interface into a pleasant one.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Non-streaming | SSE streaming |
|---|---|---|
| First visible content | After full decode (~20 s) | At TTFT (~hundreds of ms) |
| Total decode time | Same | Same |
| Transport | Single HTTP response body | Long-lived HTTP event stream |
| Connection cost | Closes quickly | Held open whole completion |
| Best fit | Short replies, batch jobs | Interactive chat and agents |
Real products, models, and research that use this idea.
- ChatGPT, Claude.ai, and Gemini all stream tokens one at a time over SSE so the reply starts rendering within a fraction of a second.
- The OpenAI and Anthropic APIs expose a stream flag that switches the response body to an SSE event stream of partial chunks.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would you choose WebSockets over SSE for an LLM application?
SSE is one-way server to client, which matches token output. Reach for WebSockets when the client must also stream upward mid-response, as in live audio or interruptible voice agents. The bidirectional channel justifies the heavier protocol only then.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming streaming makes generation faster. It does not. Throughput and total decode time are unchanged. Only the time to first visible content drops, which is a user experience win, not a speedup.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.