Glossary · Inference

Greedy Decoding

Also known as: Argmax decoding

At each step, pick the single highest-probability token. Fast and deterministic, but often loops.

The simplest decoding strategy: at each step, pick the single highest-probability next token. Fast and deterministic but prone to repetitive loops and missed globally-better completions.

In practice

The simplest baseline. Knowing why everyone uses temperature/top-p instead is foundational.

How it compares

vs. Top-p (Nucleus) Sampling →

Greedy picks the argmax; top-p samples from a dynamic nucleus of plausible tokens.

Comparisons that include Greedy Decoding

Greedy vs Beam Search vs SamplingGreedy is fastest but repetitive; beam is best for closed-ended tasks like translation; sampling with temperature and top-p is the default for open-ended generation.

Related terms

KV Cache

Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.

Temperature

Sampling knob (low = focused, high = diverse), applied to the logits before softmax.

Quantization

Run the model at lower numerical precision to save memory and accelerate inference.

FlashAttention

A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.

Grouped-Query Attention (GQA)

Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.

Knowledge Distillation

Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.