Spot the errors in this 'API is slow because of network and tokenizer' explanation
Click any words you think contain an error. Click again to unmark.
The latency is dominated by token-by-token decode, which is memory-bandwidth-bound at 50-150 tok/s per request. Network and tokenizer overhead are three orders of magnitude smaller.
Imagine ordering a 500-word letter to be written and mailed to you. Blaming the slowness on the mail truck (network) or the envelope-stuffing (tokenizer) misses the point. Those steps take seconds combined. The real cost is the writer, who can only pen one word at a time and pauses to reread their notes before each word. Five hundred words at a few words per second is several minutes of writing. The truck ride was always trivial by comparison. For an LLM, decode is that one-word-at-a-time writer, and it reads a huge cache from memory before every single token. That sequential, memory-bound writing is where almost all of the eight to twelve seconds actually goes.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Decode dominates LLM latency, and the explanation under review gets this exactly backwards. It spends its entire causal budget on network round trip and tokenizer overhead, then asserts that server-side decode is subsecond. Two of those three claims are wrong by three orders of magnitude, and the third is the opposite of the truth.
This is not a niche mistake. It is the single most common wrong diagnosis when a team first investigates why their chat product feels slow. The instinct is to blame the parts you can see in your own application code: the HTTP call, the JSON parse, the tokenizer. The part you cannot see, the token-by-token generation loop inside the provider's GPU, is where almost all of the wall clock actually lives. Because that loop is opaque, it is easy to assume it is fast and to assign the missing seconds to whatever component you happen to have a stopwatch around.
The goal of this walkthrough is to make the magnitudes concrete. By the end you should be able to estimate each cost on a napkin, explain why decode is memory-bandwidth-bound rather than compute-bound, and separate the per-request latency one user feels from the aggregate throughput a server advertises. Those two numbers differ by an order of magnitude, and conflating them is the seed of the whole misconception. We will work through the numbers for each named cost, then build the corrected latency budget from the ground up.
Putting numbers on each named cost
Start with the two costs the explanation blames. Network round trip to a major provider is single-digit to low-hundreds of milliseconds. Call it 50 to 100 milliseconds total for the request. The figure of 50 milliseconds each way is reasonable for a cross-region hop, and TLS plus connection reuse keeps the rest small. Detokenization runs in microseconds per token. Mapping a token id back to its byte string is a hash table lookup and a string append, so a 500-token response carries on the order of a single millisecond of tokenizer work, not the one to two milliseconds per token the explanation claims.
Add those up and you get well under 200 milliseconds. The observed latency is 8 to 12 seconds. The two named causes together account for under two percent of the wall clock. They are rounding error, the kind of cost you would never see in a flame graph next to the generation loop.
Now the cost the explanation dismisses. A single chat request sustains roughly 50 to 150 tokens per second. At the low end, 500 tokens take 10 seconds. At the high end, a little over 3 seconds. The observed 8 to 12 seconds sits right inside that band. The arithmetic does not merely suggest decode dominates. It accounts for essentially the entire wall clock with no other cost needed. Whenever a measured latency lines up with the decode estimate this cleanly, you have found your bottleneck and any other explanation is competing for the leftover few percent.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM serving traces show per-request decode at 50-150 tok/s while aggregate batched throughput reaches thousands per second: exactly the conflation this question targets.
- Anthropic and OpenAI both stream tokens in their APIs precisely because per-token decode latency, not network, sets the user-perceived speed.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is per-request decode capped near 50-150 tok/s while servers advertise thousands?
Separate latency from throughput. One request streams the full weights and KV cache from HBM per token, so it is bandwidth-bound and sequential. Batching many requests amortizes those reads across the batch, raising aggregate tokens per second without speeding up any single stream.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming wall clock latency on network RTT or tokenizer overhead. Both are milliseconds or microseconds. The dominant cost is sequential decode at 50-150 tokens per second per request.
60 second bullets to scan on the way to the call.
Why decode, not network or tokenizer, dominates response latency
The order of magnitude of network round trip versus decode time
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.