Walk through how top_p (nucleus) sampling truncates the distribution

Attempt it

Sort the post-softmax probabilities in descending order. Take the smallest prefix of that sorted list whose cumulative probability mass first exceeds p (e.g. 0.9). Discard everything after that prefix. Renormalize the surviving probabilities so they sum to 1, then sample a token from this truncated distribution. The cutoff is adaptive: when the model is highly confident, a few tokens already cover 90% of the mass and the nucleus is small; when the model is uncertain, the nucleus expands to include more candidates. That adaptivity is the whole point versus a fixed-K alternative.

Your answer

Quick reference

TL;DR

top_p (nucleus) sampling keeps the smallest sorted prefix of post-softmax probabilities whose cumulative mass exceeds p, renormalizes, then samples.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine the model is about to vote on the next word, and every word in the vocabulary holds up a sign showing how likely it is. Picture sorting those signs from tallest to shortest. You start adding them up from the tallest down: that one accounts for 60 percent, the next for 20, the next for 8, and so on. You stop the moment the running total crosses your threshold, say 90 percent. Only the words in that nucleus get a vote, and the rest are sent home. Then you reweight the voters so their numbers still add to 100 and pick one. The nucleus shrinks when the model is sure and grows when it is unsure, which is the whole trick.

Key concepts

Top-p nucleus sampling Sampling strategies Top-k sampling Temperature Decoding

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: four-step recipe + adaptive nucleus size + why top_p beats top_k on flat distributions + temperature/top_p/top_k order + production defaults in 2026 chat APIs.

Real-world examples

Real products, models, and research that use this idea.

GPT-5.5's chat completions endpoint exposes `top_p` as a primary sampling knob, with the default 1.0 (no truncation) for safety on production traffic.
Anthropic's Claude Sonnet 4.6 API accepts `top_p` and `temperature` together; the Messages docs warn against tuning both at once because effects compound.
Gemini 3.1 Pro on Google AI Studio surfaces `topP` in its generation config, with the recommended creative-writing setting around 0.95.
vLLM, SGLang, and TensorRT-LLM serving Llama 4 Maverick or DeepSeek V4 all implement nucleus sampling with temperature applied before top_p in the logits processor.
HuggingFace `transformers` `generate()` defaults to top_p=1.0 but documents 0.9-0.95 as the canonical creative band.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the choice of top_p interact with temperature in production?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the kernel-level cost of computing the nucleus, and can it block the decode step?

Sign in or go Pro to see the approach for this follow-up.

QWhy might you prefer min_p over top_p in some 2026 chat stacks?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating top_p as a fixed-size cutoff. The nucleus expands and contracts with the model's per-step confidence, which is exactly what top_k cannot do.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.