Greedy decoding and temperature sampling: does one cost more per token than the other?

Both greedy decoding and temperature sampling run exactly the same expensive operation per token: one full forward pass through the model, which reads the entire KV cache from HBM and computes the next-token distribution over the vocabulary. What differs is what happens after the distribution is computed. - **Greedy** takes the argmax: scan the vocabulary and pick the highest-probability token. This is one cheap pass over a 30k-300k vector, microseconds of work. - **Temperature sampling** divides the logits by a temperature value, applies softmax, then draws a token at random weighted by the resulting probabilities. Also microseconds of work. Both post-softmax operations are completely negligible next to the forward pass (which involves billions of operations). So per-token cost is essentially identical between the two strategies. Where then does any cost difference come from in production traces? From output length. Greedy and sampling walk different trajectories through token space, so the same prompt may produce a 50-token response under greedy and a 200-token response under sampling (or vice versa). The total bill is per-token-cost times number-of-tokens. The per-token-cost is the same; the token count is what differs. The practical lesson: switching from sampling to greedy is not a cost lever. If you want to spend less, reduce output length (concise prompting, max_tokens caps, brevity instructions) or use a smaller model. Tuning temperature changes output style but not per-token compute.

Greedy and temperature sampling have identical per-token compute cost. The expensive part of a decode step is the forward pass: one matmul-heavy traversal of the model that reads the KV cache from HBM, produces the hidden states, and applies the LM head to compute logits over the vocabulary. Both strategies do this work identically. The selection step that follows the LM head is what differs, and it is trivially cheap. Greedy reduces over the vocabulary to find the argmax: O(V) work on a vector of V elements (typically 30k-300k). Temperature sampling first scales the logits by 1/T (an O(V) elementwise op), then applies softmax (O(V)), then draws from the resulting categorical distribution (O(V) for the cumulative-sum trick or O(V log V) for sort-based methods). All of this is a few microseconds on a modern GPU and completely dwarfed by the forward pass (which is in the millisecond range even at small batch). So a single decode step costs essentially the same under either strategy. You cannot save meaningful money by flipping a sampling switch, and you cannot meaningfully slow down a deployment by enabling sampling. Real cost differences in production traces come from a different axis: total response length. Greedy and sampling walk different trajectories through token space. For the same prompt, the chosen tokens diverge starting from any position where the second-most-probable token had non-trivial mass. Different trajectories often produce different-length responses: the model may take a more verbose path under sampling, or hit a natural stopping boundary at a different point. The billing implication is direct. A 200-token sampling response costs 4x what an 80-token greedy response costs, despite identical per-token cost. If a team A/Bs greedy versus sampling on a route and observes greedy costing less, the cause is almost always shorter average outputs under greedy, not cheaper per-token decode. The correct way to compare decoding strategies on cost: