Why does streaming (SSE) improve perceived latency without changing total decode time?

Attempt it

Total decode time is identical with or without streaming. Why does Server-Sent Events (SSE) streaming dramatically improve the user-perceived latency of an LLM application?

Free · 2 AI evals / day

Quick reference

TL;DR

Streaming pushes each token as it is generated, so the user sees content at time-to-first-token instead of waiting for the whole reply. Total decode time is unchanged; only the perceived wait shrinks.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine asking a friend a long question. One friend stays silent, works out the entire answer in their head, and recites all of it at once after a long pause. Another friend starts talking the moment the first words come to mind and keeps going. Both take the same total time to finish, but the second feels far more responsive, because you stop waiting almost immediately. Streaming makes the model behave like that second friend. The server sends each word the instant it is ready instead of holding everything back. You start reading right away, and the rest flows in as it is produced. The actual work the model does is identical either way.

Key concepts

Token streaming Server-Sent Events Time to first token Perceived latency

Quick scan · 2–4 min

Everything important, quickly.

3 min: total time identical + perceived latency as time-to-first-visible-content + TTFT numbers + SSE versus WebSockets versus chunked transfer + connection-hold cost and batching.

Side-by-side

Aspect	Non-streaming	SSE streaming
First visible content	After full decode (~20 s)	At TTFT (~hundreds of ms)
Total decode time	Same	Same
Transport	Single HTTP response body	Long-lived HTTP event stream
Connection cost	Closes quickly	Held open whole completion
Best fit	Short replies, batch jobs	Interactive chat and agents

Real-world examples

Real products, models, and research that use this idea.

ChatGPT, Claude.ai, and Gemini all stream tokens one at a time over SSE so the reply starts rendering within a fraction of a second.
The OpenAI and Anthropic APIs expose a stream flag that switches the response body to an SSE event stream of partial chunks.
Vercel's AI SDK ships streamText and useChat helpers built on SSE to wire token streams into React UIs with almost no boilerplate.
vLLM and SGLang emit streamed token deltas while continuous batching keeps the GPU saturated across many open client connections.
GitHub Copilot streams inline completions so suggestions appear progressively rather than after the full generation finishes.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen would you choose WebSockets over SSE for an LLM application?

Sign in or go Pro to see the approach for this follow-up.

QIf perceived latency is dominated by TTFT, what should you optimize first?

Sign in or go Pro to see the approach for this follow-up.

QHow does holding connections open interact with GPU-side batching?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.