Put the steps of one round of vanilla speculative decoding in the correct order
- 1Draft model autoregressively proposes K candidate tokens t_1, ..., t_K
- 2Accepted tokens (plus the resampled correction) are appended; draft KV cache is rewound to the new tip and the round repeats
- 3For each position i = 1..K, the verifier checks whether the target's distribution accepts t_i under the speculative sampling rule
- 4Target model runs ONE forward pass over the K proposed tokens in parallel, computing logits at every position
- 5On the first rejection (or after all K accepted), the target's own distribution at that position is sampled to produce a corrected token
A cheap draft proposes K tokens, the target verifies all K in one parallel pass, accepts the matching prefix, resamples the first reject, then repeats. The acceptance rule keeps the target's distribution exact.
Imagine a slow but careful editor and a fast but sloppy assistant. The assistant guesses the next five words quickly. Instead of writing each word himself, the editor reads all five guesses at once in a single glance and checks them in order. He keeps the guesses that match what he would have written, and stops at the first one he disagrees with. There he writes his own word, throws away the rest of the guesses, and lets the assistant guess again from that point. Because the editor only ever keeps words he approves, the final text reads exactly as if he had written every word himself, just much faster when the assistant guesses well.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: the five-step round in order, why the single target pass follows the proposal, the speculative sampling accept and resample rule, why it preserves the target distribution exactly, and what sets speedup.
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with both draft model and n-gram proposers, configurable per request in 2026 deployments.
- Medusa attaches extra decoding heads to the target itself, removing the separate draft model while keeping the verify and accept loop.
- EAGLE predicts target features rather than tokens for the draft, pushing acceptance rates higher on Llama 4 class models.
- TensorRT-LLM (NVIDIA) implements speculative decoding plus Medusa heads as a production latency lever on H100 and B200.
- DeepSeek V4 uses multi-token prediction during training so its own heads can serve as a built-in draft at inference.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does accepting the matching prefix and resampling the first rejection preserve the exact target distribution?
QWhat determines the expected number of tokens accepted per round, and how does that map to wall-clock speedup?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Thinking the target runs once per proposed token, or that accepting only the matching prefix changes the output distribution. The verification step is exact, not approximate.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.