Streaming
Also known as: Token streaming, SSE streaming
Send tokens to the client as they're generated instead of buffering the full response. Much better UX.
Returning model tokens to the client as they're generated rather than waiting for the full response. Implemented via server-sent events (SSE) or chunked HTTP. Drops time-to-first-token from seconds to ~hundreds of ms.
In practice
Default UX expectation for chat apps. Interviews probe SSE vs WebSocket, backpressure, and how to interleave tool calls with streaming.
Related topics
Questions that mention this term
- Name practical UX patterns that keep reasoning model latency acceptable in production
- Why pre-norm trains stably…
- Match each LLM latency metric to what it measures
- How would you detect that your production RAG vector index has gone stale, before users notice degraded answers?
- What's the 'attention sink'…
- Explain how a layer…
Related terms
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.