Streaming
Also known as: Token streaming, SSE streaming
Send tokens to the client as they're generated instead of buffering the full response. Much better UX.
Returning model tokens to the client as they're generated rather than waiting for the full response. Implemented via server-sent events (SSE) or chunked HTTP. Drops time-to-first-token from seconds to ~hundreds of ms.
In practice
Default UX expectation for chat apps. Interviews probe SSE vs WebSocket, backpressure, and how to interleave tool calls with streaming.
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.