Why do production chat APIs almost never offer beam search as a decoding option?

Attempt it

Quick reference

TL;DR

Greedy and sampling cost roughly one forward pass per token. Beam search multiplies that by beam width and best-of-n by n, so they raise serving cost most in a memory-bound regime.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine writing one sentence at a time. Greedy decoding is writing the single best next word and moving on, one pass of thought per word. Sampling is the same effort, but you roll dice to pick among the good words. Beam search is keeping several half-finished drafts alive at once and extending all of them every step, so four drafts means four times the work and four times the scratch paper. Best-of-n is writing the whole answer from scratch n separate times and keeping your favorite, so n full passes. The lesson: greedy and sampling are cheap because they keep one draft. Beam and best-of-n are expensive because they keep many, and on a busy server that extra memory and compute is what hurts.

Key concepts

Decoding strategies Beam search Best-of-n sampling KV cache

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: classify single-path versus multi-path decoding, derive the beam-width and best-of-n multipliers, then explain why the multiplier on KV cache is what sets serving cost.

Real-world examples

Real products, models, and research that use this idea.

OpenAI and Anthropic chat APIs expose temperature and top-p but not beam search, since sampling fits memory-bound serving.
vLLM and SGLang implement best-of-n by batching n parallel sequences so the independent samples amortize HBM reads.
Llama 4 and DeepSeek V4 inference defaults ship top-p sampling rather than beam, matching RLHF-tuned generation quality.
Verifier-scored pipelines use best-of-n reranking offline, paying n times decode cost for a measurably better final answer.
Hugging Face Transformers still offers num_beams for translation and summarization, where beam's precision outweighs serving cost.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the beam-width multiplier hurt more on KV-cache memory than on FLOPs?

Sign in or go Pro to see the approach for this follow-up.

QHow can best-of-n be made cheaper to serve despite the n multiplier?

Sign in or go Pro to see the approach for this follow-up.

QHow does speculative decoding change the cost picture versus these strategies?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating all decoding strategies as equal cost. Greedy and sampling are about one pass per token; beam search multiplies by beam width and best-of-n multiplies by n, including the KV cache.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.