Glossary · Inference

KV Cache

Also known as: Key-value cache

Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.

A memory optimization in transformer inference that stores previously computed key-value pairs from self-attention layers, avoiding redundant recomputation during autoregressive token generation.

In practice

The single biggest reason long-context inference is feasible. Expect questions on cache memory math (layers × heads × dim × seqlen × 2) and paged-attention variants.

Comparisons that include KV Cache

Prompt caching vs KV cachingKV cache reuses attention within a single generation; prompt caching reuses that cache across separate requests that share a prefix. Both attack the same waste from different angles.

Practice questions

What is the KV cache in transformer inference?

Questions that mention this term

In LLM serving, what is the primary driver of end to end latency for a generation request?

Related terms

Temperature

Sampling knob (low = focused, high = diverse), applied to the logits before softmax.

Quantization

Run the model at lower numerical precision to save memory and accelerate inference.

FlashAttention

A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.

Grouped-Query Attention (GQA)

Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.

Knowledge Distillation

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.

vLLM

An open-source serving engine with PagedAttention. Much higher throughput than naive HF Transformers serving.