top_k keeps the K highest-probability tokens (fixed count); top_p keeps the smallest mass-prefix exceeding p (adaptive count). Chat models prefer top_p because it tracks confidence.
Imagine the model has 128 thousand colored marbles in a jar, each marble labeled with how likely it is to be picked. With top_k, you reach in and grab the 40 biggest no matter what. Sometimes that grabs almost-empty marbles you do not want. Sometimes it misses good ones because the jar is full of evenly sized marbles. With top_p, you say 'give me enough marbles to fill 90 percent of the jar's color total'. Then the count adapts: a few fat marbles if the jar is lopsided, many small marbles if everything is roughly equal. That adapting count is the reason chat models drifted away from top_k.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
top_k and top_p are the two foundational truncation strategies for LLM decoding. Both methods cut the post-softmax distribution before sampling, but they cut by completely different criteria, which produces opposite failure modes.
top_k: the fixed-count cutoff
top_k sorts the post-softmax distribution in descending order and keeps the highest K probabilities. The sampler discards the rest, renormalizes the survivors to sum to 1, and samples a token from the renormalized distribution.
The parameter K is a small integer (commonly 40 or 50 in older defaults, sometimes 200 as a safety cap in modern stacks). It applies identically at every decode step regardless of how the distribution looks.
What top_k does
On a tightly peaked step where the top token holds probability 0.95, top_k = 40 keeps 40 tokens. The top token's neighbors share roughly 0.05 of mass spread thinly across the next 39 slots. Sampling can land on one of those minor tokens with about 5 percent probability per step. That occasional landing produces visible output glitches.
Take a flat step where the top ten tokens each hold near 0.05 (e.g. the model is uncertain about which adjective fits). Here top_k = 40 admits all ten plausible candidates plus thirty less plausible ones. The candidate set is bigger than needed but not catastrophic. top_k can fail in the other direction when K is set too low. If K = 5 on a step with ten near-tied candidates, you arbitrarily cut five legitimate options.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | top_k | top_p (nucleus) |
|---|---|---|
| Cutoff criterion | Fixed count K | Smallest mass-prefix exceeding p |
| Set size | Always K | Floats with distribution entropy |
| Peaked step behavior | Admits noisy tail under one big peak | Shrinks to a handful of tokens |
| Flat step behavior | May chop legitimate alternatives | Expands to cover plausible candidates |
| OpenAI API default | Not offered | 1.0 (no truncation) with temperature primary |
Real products, models, and research that use this idea.
- OpenAI's Chat Completions endpoint exposes top_p (default 1) but no top_k; reasoning models accept top_p only when reasoning effort is none.
- Anthropic's Messages API accepted top_p and top_k on older Claude models, marked for advanced use only; recent Claude models reject any non-default value with a 400 error.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would you still prefer top_k over top_p in modern production?
Low-batch embedded or edge deployments where kernel simplicity matters: a fixed K means preallocated buffers and no cumulative scan. top_k also works as a hard upper bound layered on top of top_p. There K = 200 catches pathological flat steps where top_p would otherwise admit a very large set.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating top_k and top_p as interchangeable knobs. They cut by completely different criteria, and a fixed K either over-admits or under-admits depending on the step.
60 second bullets to scan on the way to the call.
Definition of top_k as fixed-cardinality truncation
Definition of top_p as adaptive mass-prefix truncation
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.