Pair each decoding strategy with its determinism and diversity profile.

Attempt it

Greedy decoding

Temperature sampling (T > 0)

Temperature = 0

Beam search

At each decode step the model produces a probability distribution over its entire vocabulary. The decoding strategy is what turns that distribution into a single chosen token. **Greedy decoding** picks the highest-probability token (the argmax) every time. The same prompt always produces the same output. It is the fastest strategy and works well when you want consistency, but it can produce repetitive, robotic text on creative tasks. **Temperature sampling** draws a token at random from the distribution. The temperature parameter scales the logits before softmax: lower temperature concentrates mass on the most likely tokens, higher temperature flattens it. Setting temperature to exactly zero collapses to greedy, because dividing by zero pushes all mass onto the argmax. Sampling is usually combined with top-k or top-p (nucleus) filters that throw away the long tail of low-probability tokens before sampling, preventing rare-token derailments. **Beam search** keeps several candidate sequences alive at once (typically 3 to 8) and extends each step by step. At the end it picks the sequence with the highest overall probability. It is more globally optimal than greedy but much slower, and rarely used for chat-style LLM serving.

Decoding strategies all start from the same place: the model's post-softmax distribution over the vocabulary at each step. They differ in how they collapse that distribution into a single chosen token. **Greedy decoding** takes the argmax. It is deterministic, fastest (no sampling overhead), and tends to produce repetitive output because the most likely token is often a function word that pulls the model into common phrases. Used for tasks where consistency matters: code completion, structured output, eval reproducibility. **Temperature sampling** divides logits by a temperature T before softmax. Lower T sharpens the distribution; higher T flattens it. With T=1 you sample from the model's native distribution. Combine with top-p (nucleus): renormalise over the smallest set of tokens whose cumulative probability exceeds p, typically 0.9 to 0.95. Or top-k: keep only the k highest-probability tokens, typically 40 to 50. Both filters cut the rare-token tail before the sampler can pick from it. Setting temperature to 0 collapses to greedy mathematically: logits divided by zero diverge, and softmax of diverging logits with a unique maximum places all mass on that single token. Production APIs implement this as a special case rather than literally dividing by zero. **Beam search** maintains a beam of B candidate sequences. Each step expands every candidate by every plausible next token, scores the resulting B times V sequences, and keeps the top B. It is more globally optimal than greedy but costs B times the compute and produces less natural output for open-ended generation, so it has fallen out of favour for chat LLMs.

Quick reference

TL;DR

Greedy picks the argmax token every step (deterministic). Temperature sampling draws stochastically (diverse). Temperature 0 collapses to greedy.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine guessing the next word in a sentence. The safe strategy is to always pick the single most likely word. You get the same answer every time, but the writing feels stiff and repetitive. The creative strategy is to roll a weighted die over several likely words and pick based on the roll. The result varies and feels more natural, but you might roll an unlucky word once in a while. A knob called temperature controls how spread out the die is. Crank it to zero and the die only shows one face, the safe choice. Raise it and the die spreads out, getting wilder. Beam search keeps several whole sentences alive in parallel and picks the best at the end, which costs more time but finds smoother sequences.

Key concepts

Greedy decoding Temperature sampling Top-p (nucleus) sampling Beam search

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define greedy as argmax, walk through temperature scaling and why T=0 collapses to greedy, contrast top-p and top-k as tail-suppression filters, position beam search in its remaining niches, then close on production settings per task type.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's GPT-5.5 API exposes temperature, top_p, and seed; setting temperature 0 and a fixed seed gives near-deterministic outputs for evals.
Anthropic's Claude Opus 4.7 uses temperature, top_p, and top_k; defaults are tuned for chat at T=1.0 with top_p=0.999.
vLLM and SGLang implement greedy as a special-case argmax to avoid divide-by-zero when temperature is exactly zero.
DeepSeek V4 reasoning mode uses self-consistency sampling: 8 samples at T=0.7, majority vote for the final answer, used in chain-of-thought prompting.
Hugging Face transformers generate API supports beam search with num_beams parameter, mostly used for translation models like MarianMT and NLLB.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does setting temperature exactly to zero collapse sampling to greedy?

Sign in or go Pro to see the approach for this follow-up.

QHow do top-p and top-k differ in failure modes?

Sign in or go Pro to see the approach for this follow-up.

QWhy has beam search fallen out of favour for chat-style LLM serving?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating temperature as quality: low equals good, high equals bad. Temperature is a creativity vs determinism knob; the right value depends on the task, and zero is wrong for creative writing.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.