Does asking for top-k logprobs add real compute cost per token, or only bytes on the wire?

Short answer·Medium·4.0 · 0·~3 min·Asked atOpenAI

Attempt it

Explain what happens at the model head when a caller requests top-k logprobs per token. Identify what is essentially free, what genuinely costs something, and how that cost scales with k and with streaming.

Free · 2 AI evals / day

**Logprobs are essentially free on the GPU.** Every decode step already runs a forward pass that produces a logit for every token in the vocabulary, then a softmax that turns those logits into probabilities. The sampler reads from that distribution to pick the next token. The distribution already exists in GPU memory regardless of whether you asked for logprobs or not. When you request top-k logprobs, the server does a small partial sort over the vocabulary (typically 30k to 300k entries) to grab the k largest. A partial sort is a handful of microseconds on a modern GPU. Compared to the matmul over billions of parameters and the KV-cache read from HBM, this is invisible. The per-token GPU latency barely moves. The real cost lives outside the forward pass. Each generated token now ships back to the caller with k extra (token, logprob) pairs. The response payload grows roughly linearly with k. On a streaming endpoint that emits one SSE event per token, every event grows, the network bandwidth grows, and the server pays CPU time to serialize those extras into JSON on every step. At high k (say, 20 or more) the serialization plus wire overhead can outweigh the GPU's per-token cost on a fast small-batch decode. That is where users actually feel the slowdown, as higher streaming latency and bandwidth, not as more GPU time. Some providers bill logprobs as output-token-equivalent precisely because the serialization and bandwidth load is real at scale.

**The compute is already paid for.** Every decode step computes a full softmax over the vocabulary because the sampler needs the distribution to pick the next token. Logprobs just expose what is already in GPU memory. Pulling the top-k is a partial sort over 30k to 300k entries, a few microseconds, invisible next to the matmul and the KV-cache read. So the GPU latency per token is essentially unchanged. Where the cost shows up is the response path. Each token now carries k extra (token, logprob) pairs. Payload size scales roughly linearly in k. On a streaming endpoint that fires one SSE event per token, the per-event payload grows, total network bytes grow, and the server pays CPU-side JSON serialization on every step instead of once at the end. The asymmetry matters. A single non-streaming response with k=10 is fine: one serialization, one fatter payload, no one notices. The same k=10 over a 500-token stream means 500 fatter SSE events, 500 serialization calls, and 500 network round-trips that each got heavier. At k=20 or higher on small-batch decode, serialization and wire time can dominate the per-token GPU step, and the user perceives the request as slower even though no extra weight got matmul'd. This is why some APIs price logprobs as an output-token-equivalent surcharge. They are not charging for compute; they are charging for the bandwidth and CPU serialization load that hides behind a feature flag.

Quick reference

TL;DR

Logprobs are near-free on the GPU because the softmax already exists; the real cost is bytes on the wire, JSON serialization per token, and amplified streaming latency.

Memory aid

Easy to grasp

Imagine a chef who already chops twenty vegetables for every dish but only puts five on the plate. If you ask to see all twenty, the chopping work was already done; the only extra effort is wrapping the extras and shipping a heavier box to your table. That is what logprobs feel like at the model head. Every time the model picks the next word, it has already produced a probability score for every possible candidate word whether you ask for them or not. Pulling out the top few is a quick partial sort. What actually costs something is packing those extra (word, score) pairs into JSON for every single streamed word and sending a fatter package over the network. The bigger your k, the heavier each package, and on a word-by-word stream that adds up to real bandwidth and packaging time.

Key concepts

Logprobs Streaming inference Softmax over vocabulary

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: softmax already exists + top-k is a free sort + payload grows linearly in k + streaming amplifies serialization + practical guidance on when to disable.

Real-world examples

Real products, models, and research that use this idea.

OpenAI Chat Completions exposes top_logprobs up to 20 and bills logprobs at an output-token-equivalent rate on most tiers, reflecting the serialization and wire load.
Anthropic Claude Opus 4.7 and Sonnet 4.6 surface logprobs through a per-token logprobs field in streamed responses, with the same per-event payload growth pattern.
Together AI and Fireworks both document logprobs as a payload-cost feature and recommend keeping k small on streaming endpoints for latency reasons.
DeepSeek V4 API returns logprobs but advises non-streaming or batched delivery when k exceeds 5, citing serialization overhead per SSE event.
Calibration tools like Inspect AI and EleutherAI's lm-eval-harness use logprobs only at evaluation time precisely because the cost profile is wrong for production serving.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhere exactly does the top-k extraction happen, and is it on the GPU or the CPU?

QHow does speculative decoding interact with logprobs requests?

QCould logprobs cost more compute if you asked for top-k over a very large k or k = vocabulary?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying logprobs cost extra GPU compute because the model has to 'compute probabilities for every token.' It already does. The softmax is part of every forward pass; logprobs are a free side effect.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.