Speculative decoding promises faster tokens: when does it actually deliver?
Describe how speculative decoding lowers decode latency and the condition under which it stops paying off.
Speculative decoding has a cheap draft model guess several tokens that the big model verifies in one pass — it only wins when acceptance is high.
Imagine a slow expert proofreading a document. Instead of writing one word at a time, a fast intern jots down the next few words they expect, and the expert glances at the whole guess at once — keeping the part that's right and fixing the first wrong word. When the intern guesses well, the expert finishes way faster because checking a sentence is barely slower than checking one word. When the intern guesses badly, the expert ends up redoing most of it, and the trick saves almost nothing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is one of the few inference tricks that is simultaneously a latency optimization, a sampling-theory result, and a hardware-utilization story — which is exactly why it shows up in senior serving interviews. The naive question is "why is generation slow?" The deeper one is "why can we cheat the sequential bottleneck without changing what the model says?"
The answer rests on a single hardware fact: an autoregressive decode step is gated by memory bandwidth, not arithmetic. Reading the weights out of HBM dominates, and scoring one position versus several costs almost the same. That slack is the resource speculation spends.
This deep dive builds the idea in layers. First the propose and verify loop and why it is lossless. Then the bandwidth argument that makes verification cheap. Then the acceptance-rate math that decides whether you actually win. Finally the production reality in 2026 — self-drafting heads, batch-size tension, and when to leave it switched off.
The propose and verify loop, step by step
Start with the baseline. Plain autoregressive decode emits one token per forward pass: feed the context, sample a token, append it, repeat. Each token is one full traversal of the model, and the traversals are strictly sequential because token t+1 depends on token t.
Speculative decoding inserts a cheap proposer. A small draft model runs its own short autoregressive loop and produces k candidate tokens — say the next 4. These are guesses about what the big target model would have generated.
The target then does the clever part. It runs a single forward pass that scores all k draft positions at once, because it already knows the candidate tokens and can lay them out as a fixed input. From that one pass it reads off, for each position, the probability it would have assigned. It accepts tokens left to right until the first one that fails an acceptance test, keeps that prefix, and resamples a corrected token at the break point.
The net result of one round is: somewhere between 1 and k+1 real tokens, produced with one draft loop plus one target pass. When the draft tracks the target well, you routinely bank 3-4 tokens per target step instead of 1.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Regime | Acceptance | Net effect |
|---|---|---|
| Predictable / in-distribution text | High (e.g. 70-90%) | Several tokens per target step; large latency cut |
| Hard / out of distribution text | Low (e.g. 20-30%) | Draft + verify cost barely amortized; little gain |
| Heavy continuous batching | Any | GPU already compute-bound; spare capacity shrinks, win fades |
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with draft-model, n-gram, and EAGLE/Medusa-style options for production serving.
- Anthropic and OpenAI use speculative-style techniques to reduce per-token latency on frontier chat models.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the rejection-sampling acceptance rule preserve the target distribution exactly?
Walk through accepting with probability min(1, p_target/p_draft) and resampling from the normalized residual on rejection.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming a fixed 2-3x speedup as if it were free, instead of tying it to the draft model's acceptance rate on the actual workload.
60 second bullets to scan on the way to the call.
Draft then verify mechanism and the single parallel verification pass
Why decode being memory bandwidth bound makes parallel verify nearly free
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.