Compare top_k and top_p as truncation strategies for sampling

**top_k** and **top_p** are both ways to throw away unlikely tokens before the sampler picks the next word, but they slice the distribution differently. top_k keeps the K highest-probability tokens, where K is a fixed number you set (commonly 40 or 50). Every decode step uses the same K. After the cut, the surviving probabilities are renormalized and sampled. top_p, also called nucleus sampling, keeps the smallest set of tokens whose cumulative probability mass first exceeds `p` (commonly 0.9). The size of that set is not fixed; it grows when the model's distribution is flat (many tokens look similar) and shrinks when the model is confident (one or two tokens dominate). Chat models converged on top_p because the fixed-K cutoff misbehaves in two opposite directions. On a confident step, K = 40 admits 39 unlikely tokens you do not want. On a flat step, K = 40 might chop off legitimate alternatives. top_p sidesteps both by letting the set size float with the distribution. GPT-5.5, Claude Sonnet 4.6, and Gemini 3.1 Pro all expose top_p as the primary truncation knob in 2026 chat APIs.

**top_k** picks by count; **top_p** picks by mass. That single sentence captures the whole comparison. top_k sorts the post-softmax distribution and keeps the highest K probabilities, renormalizes, and samples. K is the same at every step. top_p sorts the same way but instead keeps the smallest prefix whose cumulative probability exceeds `p`, then renormalizes and samples. The prefix size floats with the distribution. The failure modes are asymmetric. On a peaked step where one token holds 0.95 of the mass, top_k = 40 admits 39 garbage tokens that share 5 percent of mass; sampling occasionally emits one and the output looks broken. On a flat step where ten tokens each hold around 0.05, top_k = 40 admits all ten plus thirty less-plausible ones; the noise floor goes up. top_p handles both: on the peaked step it shrinks to one or two tokens, on the flat step it expands to cover ten or more. Chat APIs in 2026 (GPT-5.5, Claude Sonnet 4.6, Gemini 3.1 Pro) default to top_p = 1.0 and use temperature for variety control, with top_k usually optional or applied as a safety upper bound. Open-source serving stacks like vLLM and TensorRT-LLM expose both, and the canonical recipe applies temperature first, then top_p, then top_k.