Which regimes can make speculative decoding fail to speed up, or actively slow down, a serving stack?
Speculative decoding loses when acceptance is low, the draft is too costly, or the batch is already compute-bound: it converts spare FLOPs into latency, so with no spare FLOPs there is nothing to convert.
Imagine a slow expert chef plating one dish at a time. You hire a fast apprentice to guess the next few dishes ahead, and the expert just glances to confirm them in one go. If the apprentice guesses well, the expert approves a whole batch at once and you finish faster. But if the apprentice keeps guessing wrong, the expert redoes everything, so the help was wasted. And if the kitchen is already slammed cooking for a hundred tables at once, the expert has no spare moment to check guesses, so the apprentice only adds to the pile. The trick helps only when guesses are good, the apprentice is cheap, and the kitchen has idle hands to spare.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is one of the most effective latency optimizations in modern LLM serving, and also one of the most over-applied. The technique is simple to state. A small, cheap draft model proposes several tokens ahead. The large target model verifies all of them in a single parallel forward pass, accepting the longest prefix that matches what it would have produced and correcting the first mismatch. When proposals are good, you generate several tokens for roughly the cost of one target step.
The reason it works is subtle, and the reason it fails follows directly from that same reason. At small batch sizes, autoregressive decode is memory-bandwidth bound: the GPU spends most of each step streaming model weights and the KV cache out of HBM, while the arithmetic units sit largely idle. Verifying several draft tokens in parallel costs almost nothing extra because it rides on that idle compute. The technique converts spare FLOPs into lower latency.
This deep dive walks through the speedup model, then the four regimes where the trade inverts: low acceptance from high-entropy outputs, an oversized draft, domain mismatch, and the most commonly missed one, a large compute-bound batch. It closes with the fake-incompatibility distractor that staff interviewers plant to test whether a candidate reasons from the mechanism or pattern-matches on buzzwords.
The mechanism and the two levers that govern it
Each speculative cycle runs the draft model forward to propose K candidate tokens, then runs the target once over all K positions in parallel. The target's distribution at each position decides acceptance: a rejection sampling scheme accepts the longest matching prefix and resamples at the first divergence, which provably preserves the target's exact output distribution. So speculative decoding is lossless in quality; it only changes speed.
Two numbers decide whether it is faster. The acceptance rate measures how often a draft token survives verification, which tracks how closely the draft distribution matches the target. The cost ratio is the draft forward pass cost divided by the target forward pass cost.
The expected tokens accepted per cycle grow roughly geometrically as acceptance rises, while the per-cycle cost grows linearly in the cost ratio. The net speedup is the ratio of those two effects. High acceptance with a cheap draft multiplies your token rate. Push either lever the wrong way and the multiplier drops below one, at which point plain autoregressive decoding is faster.
It helps to internalise the shape of the break-even surface. For a fixed proposal length, there is a curve in acceptance-versus-cost-ratio space below which speculation is a net loss. Increasing the proposal length does not move you up that curve indefinitely: longer proposals help when acceptance is high, because more tokens clear per verification, but they hurt when acceptance is low, because a single early rejection throws away all the work behind it. So the proposal length is itself a tuned parameter, and the optimal length shrinks as acceptance falls. A serving stack that fixes one proposal length for all workloads is leaving speedup on the table in the easy regimes and bleeding it in the hard ones.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM gates speculative decoding on batch size and disables it under heavy concurrent load, since large batches are already compute bound.
- TensorRT-LLM ships both FP8 KV cache and speculative decoding together, directly contradicting the claimed incompatibility in the distractor option.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does a large decode batch flip speculative decoding from a win to a loss?
Trace the bottleneck. At small batch decode is memory bound with idle compute, which verification fills for free. At large batch the target already saturates compute, so parallel verification adds raw FLOPs rather than absorbing spare ones, and total throughput can drop.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating speculative decoding as a free win. It helps only when decode is memory bound with spare compute; at large batch the GPU is already compute saturated and verification just adds work.
60 second bullets to scan on the way to the call.
The two formula parameters that govern wins versus losses
Why high-entropy outputs collapse the acceptance rate
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.