top_k keeps the K highest-probability tokens (fixed count); top_p keeps the smallest mass-prefix exceeding p (adaptive count). Chat models prefer top_p because it tracks confidence.
Imagine the model has 128 thousand colored marbles in a jar, each marble labeled with how likely it is to be picked. With top_k, you reach in and grab the 40 biggest no matter what; sometimes that grabs almost-empty marbles you do not want, sometimes it misses good ones because the jar is full of evenly sized marbles. With top_p, you say 'give me enough marbles to fill 90 percent of the jar's color total' and the count adapts: a few fat marbles if the jar is lopsided, many small marbles if everything is roughly equal. That adapting count is the reason chat models drifted away from top_k.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
top_k and top_p are the two foundational truncation strategies for LLM decoding, and the choice between them is one of the most common sampling-parameter questions in interviews. Both methods cut the post-softmax distribution before sampling, but they cut by completely different criteria, which produces opposite failure modes.
This deep dive walks through the mechanics of each method, why their failure modes are asymmetric, why production chat models converged on top_p, where top_k still earns its keep, and how the two compose with temperature and the newer min_p variant.
top_k: the fixed-count cutoff
top_k sorts the post-softmax distribution in descending order and keeps the highest K probabilities. Whatever is left over is discarded, the survivors are renormalized to sum to 1, and a token is sampled from the renormalized distribution.
The parameter K is a small integer (commonly 40 or 50 in older defaults, sometimes 200 as a safety cap in modern stacks). It applies identically at every decode step regardless of how the distribution looks.
What top_k actually does
On a tightly peaked step where the top token holds probability 0.95, top_k = 40 keeps 40 tokens. The top token's neighbors share roughly 0.05 of mass spread thinly across the next 39 slots. Sampling can land on one of those minor tokens with about 5 percent probability per step, and that occasional landing produces visible output glitches.
On a flat step where the top ten tokens each hold near 0.05 (e.g. the model is genuinely uncertain about which adjective fits), top_k = 40 admits all ten plausible candidates plus thirty less plausible ones. The candidate set is bigger than needed but not catastrophic. Where top_k can clearly fail in the other direction is K set too low: if K = 5 on a step with ten near-tied candidates, you arbitrarily cut five legitimate options.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | top_k | top_p (nucleus) |
|---|---|---|
| Cutoff criterion | Fixed count K | Smallest mass-prefix exceeding p |
| Set size | Always K | Floats with distribution entropy |
| Peaked step behavior | Admits noisy tail under one big peak | Shrinks to a handful of tokens |
| Flat step behavior | May chop legitimate alternatives | Expands to cover plausible candidates |
| 2026 chat API default | Off or high (200+) | 1.0 (no truncation) with temperature primary |
Real products, models, and research that use this idea.
- GPT-5.5's chat completions endpoint exposes both `top_p` (default 1.0) and previously hidden `top_logprobs`; top_k is not surfaced.
- Anthropic's Claude Sonnet 4.6 Messages API accepts `top_p` and `top_k` as separate parameters, with docs advising against tuning both at once.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would you still prefer top_k over top_p in 2026 production?
Low-batch embedded or edge deployments where kernel simplicity matters: a fixed K means preallocated buffers and no cumulative scan. Also as a hard upper bound layered on top of top_p, where K = 200 catches pathological flat steps that top_p would otherwise blow open.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating top_k and top_p as interchangeable knobs. They cut by completely different criteria, and a fixed K either over-admits or under-admits depending on the step.
60 second bullets to scan on the way to the call.
Definition of top_k as fixed-cardinality truncation
Definition of top_p as adaptive mass-prefix truncation
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.