Explain what happens at the model head when a caller requests top-k logprobs per token. Identify what is essentially free, what genuinely costs something, and how that cost scales with k and with streaming.
Logprobs are near-free on the GPU because the logits already exist. The real cost is bytes on the wire, JSON serialization per token, and added streaming latency.
Imagine a chef who already chops twenty vegetables for every dish but only puts five on the plate. If you ask to see all twenty, the chopping work was already done. The only extra effort is wrapping the extras and shipping a heavier box to your table. That is what logprobs feel like at the model head. Every time the model picks the next word, it has already produced a probability score for every possible candidate word whether you ask for them or not. Pulling out the top few is a quick partial sort. What costs something is packing those extra (word, score) pairs into JSON for every streamed word. Sending that larger package over the network costs something too. The bigger your k, the heavier each package, and on a word-by-word stream that adds up to real bandwidth and packaging time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Logprobs add almost no GPU compute, because the forward pass already produces a logit for every vocabulary token at each decode step. Their cost on a production system can still be large. The work moves from the GPU, which has plenty of spare capacity, to the CPU and the network, which often do not.
To reason about this cost, find where the bottleneck sits instead of counting GPU FLOPs. The cost has three layers: compute is nearly free, serialization is real, and streaming repeats the serialization for every token. The claim that the model must do extra work to compute probabilities is wrong.
What the forward pass already produces
Every decoder forward pass ends with a head that projects the final hidden state through a lm_head matrix to produce a logit vector of size V, the vocabulary. V is 32k for Llama 2 and Mistral 7B, about 128k for Llama 3, 50k to 200k for GPT-family tokenizers, and up to a few hundred thousand for some multilingual models. When the server samples, a softmax turns that vector into a probability distribution over the next token.
The sampler needs those scores to pick a token. Greedy decoding reads the argmax of the logits and needs no softmax. Temperature, top-k and top-p sampling compute a softmax over the scaled logits. Either way, the full logit vector is computed on every step.
Returning logprobs to the caller adds only a little GPU work. The server forwards information that already exists in device memory. The new operations are one log-softmax over the V logits and a top-k selection over V entries, and GPU kernels run both in microseconds. Compared to the matmul cost (billions of parameter multiplies) and the HBM traffic (gigabytes per token), that time is negligible.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI Chat Completions returns per-token logprobs when logprobs is true, with top_logprobs set from 0 to 20 alternatives per position.
- The Anthropic Messages API returns no token log probabilities, so Claude-based apps estimate confidence another way, for example by sampling several answers or asking for a score.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere exactly does the top-k extraction happen, and is it on the GPU or the CPU?
In vLLM and TensorRT-LLM the top-k selection runs on the GPU, over the logit tensor in device memory. The vLLM sampler uses a log-softmax and a torch.topk call. Only the top-k values and token IDs are copied to the host, so the GPU-to-CPU transfer stays small.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying logprobs cost extra GPU compute because the model has to 'compute probabilities for every token.' It already scores every token. Every forward pass produces logits over the full vocabulary, so logprobs add almost no GPU work.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.