Compare top_k and top_p as truncation strategies for sampling

Attempt it

Both restrict the post-softmax distribution to a subset of tokens and then renormalize before sampling, but they pick the subset differently. top_k keeps the K highest-probability tokens, a fixed-size cutoff regardless of distribution shape; the same K is applied at every step. top_p (nucleus) keeps the smallest set of tokens whose cumulative probability exceeds p, an adaptive cutoff that grows when the model is uncertain and shrinks when it is confident. Chat models largely moved to top_p because the fixed K of top_k either includes garbage tail tokens when the model is confident (K too large) or chops off plausible alternatives when the model is uncertain (K too small). top_p adapts to the model's per-step confidence.

Your answer

Quick reference

TL;DR

top_k keeps the K highest-probability tokens (fixed count); top_p keeps the smallest mass-prefix exceeding p (adaptive count). Chat models prefer top_p because it tracks confidence.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine the model has 128 thousand colored marbles in a jar, each marble labeled with how likely it is to be picked. With top_k, you reach in and grab the 40 biggest no matter what; sometimes that grabs almost-empty marbles you do not want, sometimes it misses good ones because the jar is full of evenly sized marbles. With top_p, you say 'give me enough marbles to fill 90 percent of the jar's color total' and the count adapts: a few fat marbles if the jar is lopsided, many small marbles if everything is roughly equal. That adapting count is the reason chat models drifted away from top_k.

Key concepts

Top-k sampling Top-p nucleus sampling Sampling strategies Temperature Decoding

Quick scan · 2–4 min

Everything important, quickly.

3 min: top_k as count cutoff + top_p as mass cutoff + peaked vs flat failure modes + 2026 defaults + temperature/top_p/top_k order + min_p as the newer variant.

Side-by-side

Property	top_k	top_p (nucleus)
Cutoff criterion	Fixed count K	Smallest mass-prefix exceeding p
Set size	Always K	Floats with distribution entropy
Peaked step behavior	Admits noisy tail under one big peak	Shrinks to a handful of tokens
Flat step behavior	May chop legitimate alternatives	Expands to cover plausible candidates
2026 chat API default	Off or high (200+)	1.0 (no truncation) with temperature primary

Real-world examples

Real products, models, and research that use this idea.

GPT-5.5's chat completions endpoint exposes both `top_p` (default 1.0) and previously hidden `top_logprobs`; top_k is not surfaced.
Anthropic's Claude Sonnet 4.6 Messages API accepts `top_p` and `top_k` as separate parameters, with docs advising against tuning both at once.
Gemini 3.1 Pro on Google AI Studio exposes both `topP` and `topK` in its generationConfig, defaulting topK to 40 and topP to 0.95.
vLLM's SamplingParams accepts `top_p`, `top_k`, and `min_p` all at once, applying them in the canonical order: temperature, top_p, top_k.
Llama 4 Maverick deployments via SGLang often run `temperature=0.7, top_p=0.9, top_k=200` as a balanced creative-chat default.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen would you still prefer top_k over top_p in 2026 production?

Sign in or go Pro to see the approach for this follow-up.

QHow do temperature and top_p interact, and which should you tune first?

Sign in or go Pro to see the approach for this follow-up.

QWhat is min_p and why are some 2026 stacks adopting it?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.