Streaming
Also known as: Token streaming, SSE streaming
Send tokens to the client as they're generated instead of buffering the full response. Much better UX.
Returning model tokens to the client as they're generated rather than waiting for the full response. Implemented via server-sent events (SSE) or chunked HTTP. Drops time-to-first-token from seconds to ~hundreds of ms.
In practice
Default UX expectation for chat apps. Interviews probe SSE vs WebSocket, backpressure, and how to interleave tool calls with streaming.
Related topics
Questions that mention this term
- Order the lifecycle of a single extended thinking API call
- What's the 'attention sink'…
- Describe the W_O projection in multi-head attention, its shape and what it mixes.
- Pre-norm versus post-norm: which placement makes deep stacks stable?
- Interpret what the logit lens reveals when you unembed intermediate layers
- Explain how a layer…
Related terms
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.