Walk through how top_p (nucleus) sampling truncates the distribution
top_p (nucleus) sampling keeps the smallest sorted prefix of post-softmax probabilities whose cumulative mass exceeds p, renormalizes, then samples.
Imagine the model is about to vote on the next word, and every word in the vocabulary holds up a sign showing how likely it is. Picture sorting those signs from tallest to shortest. You start adding them up from the tallest down: that one accounts for 60 percent, the next for 20, the next for 8, and so on. You stop the moment the running total crosses your threshold, say 90 percent. Only the words in that nucleus get a vote, and the rest are sent home. Then you reweight the voters so their numbers still add to 100 and pick one. The nucleus shrinks when the model is sure and grows when it is unsure, which is the whole trick.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
top_p, also called nucleus sampling, is the dominant truncation strategy for chat-style LLM decoding in 2026. Every major chat API (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) exposes it as a primary sampling knob, every open-source serving stack implements it, and most production deployments leave it near 1.0 unless they have a specific reason to clamp creativity.
This deep dive walks through the exact algorithm, why an adaptive cutoff matters versus the fixed-K alternative, the order of operations when combined with temperature, and the kernel-level performance picture on modern GPUs.
The four-step recipe in detail
Start with the post-softmax distribution over the vocabulary. For a 128k-token tokenizer, this is a 128k-element probability vector that sums to 1.
- Sort the probabilities in descending order. Token indices come along for the ride so you can map back at the end.
- Accumulate the sorted probabilities from index 0 forward. Stop at the smallest index k such that the cumulative sum first exceeds
p(commonly 0.9 or 0.95). - Cut everything after index k. The retained set is the nucleus.
- Renormalize the surviving probabilities so they sum to 1, then sample one token from the renormalized distribution.
The parameter p is the only knob. Note that the nucleus always contains at least one token (the top one) because any single probability is at most 1, which already satisfies the cumulative-mass condition once p is reached or exceeded.
A worked example
Say the sorted probabilities are [0.7, 0.15, 0.08, 0.04, 0.02, 0.01] and p = 0.9. Cumulative sums are [0.7, 0.85, 0.93, ...]. The nucleus is the first three tokens because 0.93 is the first cumulative value to cross 0.9. Renormalize to [0.753, 0.161, 0.086] and sample.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- GPT-5.5's chat completions endpoint exposes `top_p` as a primary sampling knob, with the default 1.0 (no truncation) for safety on production traffic.
- Anthropic's Claude Sonnet 4.6 API accepts `top_p` and `temperature` together; the Messages docs warn against tuning both at once because effects compound.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the choice of top_p interact with temperature in production?
Temperature reshapes the softmax (T < 1 sharpens, T > 1 flattens), then top_p truncates the reshaped distribution. Tuning both at once changes behavior non-linearly. Most teams pick one as the primary knob: lower top_p for safety, higher temperature for variety. Anthropic's docs explicitly warn against tuning both simultaneously.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating top_p as a fixed-size cutoff. The nucleus expands and contracts with the model's per-step confidence, which is exactly what top_k cannot do.
60 second bullets to scan on the way to the call.
Four-step recipe: sort, accumulate, cut, renormalize, sample
Where the parameter p enters and what it bounds
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.