Speculative decoding promises faster tokens: when does it deliver?
Describe how speculative decoding lowers decode latency and the condition under which it stops paying off.
Speculative decoding has a cheap draft model guess several tokens that the big model verifies in one pass. It only wins when acceptance is high.
Imagine a slow expert proofreading a document. Instead of writing one word at a time, a fast intern jots down the next few words they expect. The expert looks at the whole guess at once. The expert keeps the part that's right and fixes the first wrong word. When the intern guesses well, the expert finishes much faster because checking a sentence is barely slower than checking one word. When the intern guesses badly, the expert ends up redoing most of it, and the trick saves almost nothing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is one of the few inference tricks that is simultaneously a latency optimization, a sampling-theory result, and a hardware-utilization story. The naive question is "why is generation slow?" The deeper one is "why can we get around the sequential bottleneck without changing what the model says?"
The answer rests on a single hardware fact: an autoregressive decode step is limited by memory bandwidth, not arithmetic. Reading the weights out of HBM dominates, and scoring one position versus several costs almost the same. That spare capacity is the resource speculation uses.
The propose and verify loop, step by step
Start with the baseline. Plain autoregressive decode emits one token per forward pass: feed the context, sample a token, append it, repeat. Each token is one full traversal of the model, and the traversals are strictly sequential because token t+1 depends on token t.
Speculative decoding inserts a cheap proposer. A small draft model runs its own short autoregressive loop and produces k candidate tokens, say the next 4. These are guesses about what the big target model would have generated.
The target then does the clever part. It runs a single forward pass that scores all k draft positions at once. It can do this because it already knows the candidate tokens and can lay them out as a fixed input. From that one pass it reads off, for each position, the probability it would have assigned. It accepts tokens left to right until the first one that fails an acceptance test. It keeps that prefix and resamples a corrected token at the break point.
The net result of one round is: somewhere between 1 and k+1 real tokens, produced with one draft loop plus one target pass. When the draft tracks the target well, you routinely get 3-4 tokens per target step instead of 1.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Regime | Acceptance | Net effect |
|---|---|---|
| Predictable / in-distribution text | High (e.g. 70-90%) | Several tokens per target step; large latency cut |
| Hard / out of distribution text | Low (e.g. 20-30%) | Draft + verify cost barely amortized; little gain |
| Heavy continuous batching | Any | GPU already compute-bound; spare capacity shrinks, win fades |
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with draft model, n-gram, EAGLE and MTP options for production serving.
- Serving stacks use speculative decoding to cut per-token latency on chat models without changing their outputs.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the rejection-sampling acceptance rule preserve the target distribution exactly?
Walk through accepting with probability min(1, p_target/p_draft) and resampling from the normalized residual on rejection.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming a fixed 2-3x speedup as if it were free, instead of tying it to the draft model's acceptance rate on the actual workload.
60 second bullets to scan on the way to the call.
Draft then verify mechanism and the single parallel verification pass
Why decode being memory bandwidth bound makes parallel verify nearly free
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.