Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
**LLM inference latency** has two phases. Prefill reads your whole prompt in one parallel pass and is fast. Decode generates the answer one token at a time, and each step waits for the previous one. Decode is where the time goes.
The explanation under review blames three things: network round trip, tokenizer overhead, and a fast server. Two of those numbers are wildly off, and the third is backwards. Network round trip to a major provider is tens of milliseconds total, not seconds. Tokenization is microseconds per token, far below anything you would notice.
The real bottleneck is decode itself. A single chat request gets roughly 50 to 150 tokens per second, because the model must read its large key-value cache from GPU memory before producing each new token. That memory traffic, not arithmetic, sets the pace.
Do the math: 500 tokens at 50 tokens per second is 10 seconds. That matches the observed 8 to 12 seconds almost exactly. So decode accounts for essentially all of the wall clock, and the network plus tokenizer story is the classic wrong diagnosis.
**Decode dominates.** The explanation pins the latency on network and tokenizer, but both are off by orders of magnitude, and the claim that decode is subsecond is simply false.
Network round trip to a major provider is 50 to 100 milliseconds total. Tokenization runs in microseconds per token. Neither registers against an 8 to 12 second wall clock. They are rounding error, not the root cause.
The true bottleneck is autoregressive decode. Generation happens one token at a time, sequentially. Each step must stream the model weights and the growing key-value cache from HBM. So per-request throughput is bounded by memory bandwidth, not compute. A single chat request realistically sustains 50 to 150 tokens per second.
Run the arithmetic. Five hundred tokens at 50 tokens per second is 10 seconds. That lands inside the observed range. The thousands-per-second figure describes aggregate server throughput across a large batch, not the latency one user sees.
The diagnosis to give: decode is memory-bandwidth-bound and inherently sequential. Fix it with faster generation, not faster networking. Speculative decoding, larger batches, and shorter outputs move the needle. Shaving 50 milliseconds off the network does nothing.
Quick reference
TL;DR
The latency is dominated by token-by-token decode, which is memory-bandwidth-bound at 50-150 tok/s per request. Network and tokenizer overhead are three orders of magnitude smaller.
Imagine ordering a 500-word letter to be written and mailed to you. Blaming the slowness on the mail truck (network) or the envelope-stuffing (tokenizer) misses the point. Those steps take seconds combined. The real cost is the writer, who can only pen one word at a time and pauses to reread their notes before each word. Five hundred words at a few words per second is several minutes of writing. The truck ride was always trivial by comparison. For an LLM, decode is that one-word-at-a-time writer, and it reads a huge cache from memory before every single token. That sequential, memory-bound writing is where almost all of the eight to twelve seconds actually goes.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very short responses of a handful of tokens on a nearby endpoint When output is tiny, fixed network round trip really can be a large fraction of the wall clock.
Deployments behind slow or congested links, or cross-region calls Real network pathology can add hundreds of milliseconds, so measure before assuming decode dominates.
Workloads dominated by huge prompts and tiny outputs Prefill cost scales with input length, so a 100k token prompt can outweigh a short decode.
Pipelines with heavy pre or post processing around the model call Retrieval, reranking, or guardrail passes can add seconds that are unrelated to decode itself.
Quick scan · 2–4 min
Everything important, quickly.
3 min: order of magnitude each cost (network ms, tokenizer microseconds, decode seconds) + why decode is bandwidth-bound + latency versus throughput + the tokens divided by tok/s estimate + real levers.
Real-world examples
Real products, models, and research that use this idea.
1vLLM serving traces show per-request decode at 50-150 tok/s while aggregate batched throughput reaches thousands per second: exactly the conflation this question targets.
2Anthropic and OpenAI both stream tokens in their APIs precisely because per-token decode latency, not network, sets the user-perceived speed.
3DeepSeek V4 and Llama 4 ship speculative decoding and GQA to attack the sequential, bandwidth-bound decode cost rather than network overhead.
4NVIDIA TensorRT-LLM benchmarks report tokens-per-second-per-request separately from aggregate throughput on H100 and B200 to avoid this exact mistake.
5Groq markets its LPU hardware around raising single-stream decode tok/s, a direct acknowledgment that decode bandwidth, not network, is the bottleneck.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is per-request decode capped near 50-150 tok/s while servers advertise thousands?
Sign in or go Pro to see the approach for this follow-up.
QIf decode is memory-bandwidth-bound, why does adding more compute not help?
Sign in or go Pro to see the approach for this follow-up.
QWhich optimizations actually reduce the 10-second decode here?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Blaming wall clock latency on network RTT or tokenizer overhead. Both are milliseconds or microseconds. The dominant cost is sequential decode at 50-150 tokens per second per request.