vLLM
Also known as: PagedAttention
An open-source serving engine with PagedAttention. Much higher throughput than naive HF Transformers serving.
A high-throughput LLM serving engine built around PagedAttention, a KV-cache management scheme inspired by OS virtual memory paging. Serves significantly more concurrent requests than naive batching on the same hardware.
In practice
Default open-source choice for self-hosted inference. Senior infra interviews dig into PagedAttention vs the KV cache memory math.
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.