Does asking for top-k logprobs add real compute cost per token, or only bytes on the wire?

**Logprobs are essentially free on the GPU.** Every decode step already runs a forward pass that produces a logit for every token in the vocabulary, then a softmax that turns those logits into probabilities. The sampler reads from that distribution to pick the next token. The distribution already exists in GPU memory regardless of whether you asked for logprobs or not. When you request top-k logprobs, the server does a small partial sort over the vocabulary (typically 30k to 300k entries) to grab the k largest. A partial sort is a handful of microseconds on a modern GPU. Compared to the matmul over billions of parameters and the KV-cache read from HBM, this is invisible. The per-token GPU latency barely moves. The real cost lives outside the forward pass. Each generated token now ships back to the caller with k extra (token, logprob) pairs. The response payload grows roughly linearly with k. On a streaming endpoint that emits one SSE event per token, every event grows, the network bandwidth grows, and the server pays CPU time to serialize those extras into JSON on every step. At high k (say, 20 or more) the serialization plus wire overhead can outweigh the GPU's per-token cost on a fast small-batch decode. That is where users actually feel the slowdown, as higher streaming latency and bandwidth, not as more GPU time. Some providers bill logprobs as output-token-equivalent precisely because the serialization and bandwidth load is real at scale.

**The compute is already paid for.** Every decode step computes a full softmax over the vocabulary because the sampler needs the distribution to pick the next token. Logprobs just expose what is already in GPU memory. Pulling the top-k is a partial sort over 30k to 300k entries, a few microseconds, invisible next to the matmul and the KV-cache read. So the GPU latency per token is essentially unchanged. Where the cost shows up is the response path. Each token now carries k extra (token, logprob) pairs. Payload size scales roughly linearly in k. On a streaming endpoint that fires one SSE event per token, the per-event payload grows, total network bytes grow, and the server pays CPU-side JSON serialization on every step instead of once at the end. The asymmetry matters. A single non-streaming response with k=10 is fine: one serialization, one fatter payload, no one notices. The same k=10 over a 500-token stream means 500 fatter SSE events, 500 serialization calls, and 500 network round-trips that each got heavier. At k=20 or higher on small-batch decode, serialization and wire time can dominate the per-token GPU step, and the user perceives the request as slower even though no extra weight got matmul'd. This is why some APIs price logprobs as an output-token-equivalent surcharge. They are not charging for compute; they are charging for the bandwidth and CPU serialization load that hides behind a feature flag.