Why do production chat APIs almost never offer beam search as a decoding option?
Greedy and sampling cost roughly one forward pass per token. Beam search multiplies that by beam width and best-of-n by n, so they raise serving cost most in a memory-bound regime.
Imagine writing one sentence at a time. Greedy decoding is writing the single best next word and moving on, one pass of thought per word. Sampling is the same effort, but you roll dice to pick among the good words. Beam search is keeping several half-finished drafts alive at once and extending all of them every step, so four drafts means four times the work and four times the scratch paper. Best-of-n is writing the whole answer from scratch n separate times and keeping your favorite, so n full passes. The lesson: greedy and sampling are cheap because they keep one draft. Beam and best-of-n are expensive because they keep many, and on a busy server that extra memory and compute is what hurts.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: classify single-path versus multi-path decoding, derive the beam-width and best-of-n multipliers, then explain why the multiplier on KV cache is what sets serving cost.
Real products, models, and research that use this idea.
- OpenAI and Anthropic chat APIs expose temperature and top-p but not beam search, since sampling fits memory-bound serving.
- vLLM and SGLang implement best-of-n by batching n parallel sequences so the independent samples amortize HBM reads.
- Llama 4 and DeepSeek V4 inference defaults ship top-p sampling rather than beam, matching RLHF-tuned generation quality.
- Verifier-scored pipelines use best-of-n reranking offline, paying n times decode cost for a measurably better final answer.
- Hugging Face Transformers still offers num_beams for translation and summarization, where beam's precision outweighs serving cost.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the beam-width multiplier hurt more on KV-cache memory than on FLOPs?
QHow can best-of-n be made cheaper to serve despite the n multiplier?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating all decoding strategies as equal cost. Greedy and sampling are about one pass per token; beam search multiplies by beam width and best-of-n multiplies by n, including the KV cache.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.