Compare greedy, sampling and beam search at inference: cost and best use case.

Every decoding strategy runs the same transformer forward pass to produce a probability distribution over the next token. What differs is how you turn that distribution into a chosen token, and how many candidate sequences you keep alive while doing it. **Greedy decoding** takes the single highest-probability token every step. It runs one forward pass per token and keeps one KV cache, so its cost is the 1x baseline. The output is deterministic, which is great for tests and strict code generation. Sampling, including temperature and top-p, draws a token randomly from the distribution instead of taking the argmax. The forward pass is identical to greedy, so the compute and cache cost are also 1x. The only extra work is a cheap random draw. This is what chat uses, because a little randomness makes replies feel natural and varied. Beam search is the expensive one. It keeps several candidate sequences alive at once, extends each, and prunes back to the best few. With width B you run roughly B model passes per step and hold B separate caches, so both compute and memory scale with B. That suits low-creativity tasks like translation but is too costly for serving open-ended chat at scale.

All three share the same per-token forward pass. The cost difference is how many sequences each keeps alive. **Greedy** picks the argmax token, runs one pass, and is deterministic. Sampling draws from the same distribution. Its compute and cache cost match greedy exactly, plus a negligible random draw. Both are 1x baseline cost. Greedy suits reproducible tasks like tests. Sampling suits chat, where diversity helps. Beam search at width B keeps B hypotheses alive. Each beam needs its own KV cache, so memory scales by B. Each step extends all beams, so decode compute also scales by B. It once dominated translation and summarization. There, low-entropy targets reward searching for the highest-probability sequence. Chat APIs dropped beam for three reasons. First, open-ended chat is high entropy. Beam collapses toward bland, repetitive text. Second, the Bx KV memory wrecks serving throughput in a bandwidth-bound regime. Third, RLHF already shapes the distribution so a single sample reads well. Best-of-n and self-consistency, which sample n full outputs, cost n times but stay parallel and cache-cheap per stream.