Glossary · Inference

Beam Search

Also known as: Beam decoding

Keep the K best partial sequences at each step; deterministic, breadth-first decoding.

A deterministic decoding strategy that maintains the top-K highest-probability partial sequences (beams) at each step. Trades exploration for breadth; often produces bland or repetitive output for open-ended generation.

In practice

Standard for translation/summarization, rarely for chat. Interviews probe why temperature sampling won for open-ended generation.

How it compares

vs. Temperature →

Beam search is deterministic and breadth-first; temperature sampling is stochastic and single-path.

Comparisons that include Beam Search

Greedy vs Beam Search vs SamplingGreedy is fastest but repetitive; beam is best for closed-ended tasks like translation; sampling with temperature and top-p is the default for open-ended generation.

Related terms

KV Cache

Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.

Temperature

Sampling knob (low = focused, high = diverse), applied to the logits before softmax.

Quantization

Run the model at lower numerical precision to save memory and accelerate inference.

FlashAttention

A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.

Grouped-Query Attention (GQA)

Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.

Knowledge Distillation

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.