Drag each answer to line up with its matching prompt
Greedy decoding
Draws stochastically from the post-softmax distribution: diverse output, controllable via temperature/top_p/top_k
Temperature sampling (T > 0)
Keeps the top-k partial sequences each step: more deterministic than sampling, more globally optimal than greedy, costlier to compute
Temperature = 0
Collapses to greedy because dividing logits by zero pushes all mass onto the argmax token
Beam search
Picks argmax of the next token distribution every step: deterministic, fastest, most repetitive output
Greedy picks the argmax token every step (deterministic). Temperature sampling draws stochastically (diverse). Temperature 0 collapses to greedy.
Imagine guessing the next word in a sentence. The safe strategy is to always pick the single most likely word. You get the same answer every time, but the writing feels stiff and repetitive. The creative strategy is to roll a weighted die over several likely words and pick based on the roll. The result varies and feels more natural, but you might roll an unlucky word once in a while. A knob called temperature controls how spread out the die is. Crank it to zero and the die only shows one face, the safe choice. Raise it and the die spreads out, getting wilder. Beam search keeps several whole sentences alive in parallel and picks the best at the end, which costs more time but finds smoother sequences.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Decoding strategy is the final stage of every LLM forward pass. The model produces a probability distribution over its vocabulary at each step; the decoding strategy is what turns that distribution into the single token that joins the output. Get this wrong and even a perfectly trained model produces unusable text.
This deep dive separates four strategies that show up in every serving system: greedy, temperature sampling, top-p and top-k filtering, and beam search. The goal is not just to know what each one does, but to know when to use which, what failure modes each invites, and what the modern defaults are for code, chat, reasoning, and structured output tasks.
The payoff for understanding this well is concrete. Most LLM application bugs that look like model problems are actually decoding-config problems. A JSON output that occasionally adds a stray character is sampling at T=0.7 instead of greedy. A creative writing app that feels lifeless is greedy instead of T=1.0 with top-p. An eval that gives different scores on rerun forgot to fix a seed.
The rest of this section covers the mechanics of each strategy, the math of temperature including the T=0 limit, the role of top-p and top-k filters, the remaining niches for beam search, and a production settings cheat sheet.
Greedy: deterministic argmax
Greedy decoding is the simplest possible strategy. At each step, the model outputs logits over the vocabulary, softmax produces probabilities, and greedy selects the token with the highest probability. The same prompt with the same model weights always produces the same output. There is no random component.
The computational cost is minimal: just an argmax over the vocabulary tensor. No sampling, no filtering, no extra kernels. This makes greedy slightly faster than sampling in throughput-sensitive serving, though the difference is small relative to the matmul cost.
The quality trade-off is that greedy tends to produce repetitive text. The most likely next token is often a common function word, and once the model commits to that token it can fall into common phrase patterns that loop. This is why greedy is rarely used for open-ended chat, but it is the right choice for tasks where consistency matters more than variation: structured output (JSON, SQL, code), eval pipelines that need reproducibility, function calls where the model must match a schema exactly.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's GPT-5.5 API exposes temperature, top_p, and seed; setting temperature 0 and a fixed seed gives near-deterministic outputs for evals.
- Anthropic's Claude Opus 4.7 uses temperature, top_p, and top_k; defaults are tuned for chat at T=1.0 with top_p=0.999.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does setting temperature exactly to zero collapse sampling to greedy?
Walk through the math: softmax of (l/T) as T approaches zero pushes the largest logit's exponential to dominate. In the limit, the probability of the argmax token becomes 1 and all others become 0. Sampling from a distribution with a single non-zero entry is deterministic. Real implementations special-case T=0 to argmax to avoid numerical issues.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating temperature as quality: low equals good, high equals bad. Temperature is a creativity vs determinism knob; the right value depends on the task, and zero is wrong for creative writing.
60 second bullets to scan on the way to the call.
The definition of greedy decoding as argmax of the post-softmax distribution
How temperature scales logits and what T=0 means mathematically
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.