Temperature
Also known as: Sampling temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
A sampling parameter that controls the randomness of LLM outputs. Low temperature (0.0-0.3) produces deterministic, focused responses; high temperature (0.7-1.0+) produces more creative, diverse outputs.
In practice
First knob to reach for when output is too random or too repetitive. Expect questions on its interaction with top-p and top-k.
Related topics
Practice questions
Questions that mention this term
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.
vLLM
An open-source serving engine with PagedAttention. Much higher throughput than naive HF Transformers serving.