How does vanilla speculative decoding work and why is verification a single forward pass?
Walk through one round of vanilla speculative decoding. Be specific about (a) why the target's verification is a single forward pass even though K tokens are being checked, and (b) what guarantees the final distribution is identical to running the target alone.
A cheap draft proposes K tokens; the target verifies all K in one parallel forward pass; a rejection-sampling rule makes the output distribution exactly match the target alone.
Imagine a fast but sloppy assistant who guesses the next several words of your sentence while you, the careful editor, are still thinking. The editor reads all the guesses at once in a single glance instead of word by word. Wherever a guess matches what the editor would have written, the editor keeps it for free. At the first guess the editor disagrees with, the editor crosses it out, writes the correct word, and throws away every guess after it. Because the editor still makes the final call on each word using a fair coin-flip rule, the finished sentence reads exactly as if the editor wrote every word alone. The only thing the assistant changed is speed, never the final wording.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: draft proposes K tokens + target single-pass verification via causal mask + acceptance ratio rule + residual resample on rejection + the lossless equivalence + memory-bound speedup intuition.
# One round of vanilla speculative decoding (illustrative)
draft_tokens, p_draft = draft.propose(prefix, k=K) # K cheap autoregressive steps
p_target = target.forward(prefix + draft_tokens) # ONE pass, logits at all K positions
accepted = []
for i, t in enumerate(draft_tokens):
r = random.random()
if r < min(1.0, p_target[i][t] / p_draft[i][t]):
accepted.append(t) # accept draft token
else:
residual = relu(p_target[i] - p_draft[i]) # max(0, p_t - p_d)
accepted.append(sample(residual / residual.sum()))
break # discard positions after j
else:
accepted.append(sample(p_target[K])) # bonus token when all K accepted| Aspect | Standard autoregressive decode | Speculative decoding |
|---|---|---|
| Target passes per token block | One target pass per single token | One target pass verifies up to K tokens |
| Output distribution | Target distribution | Exactly the target distribution (lossless) |
| Extra compute | None | K cheap draft passes per round |
| Speedup source | None | More tokens advanced per memory-bound weight read |
| Failure mode | Latency floor at one token per pass | Low acceptance wastes draft work and verification |
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with draft model, n-gram, and EAGLE-style methods, letting operators trade acceptance rate against draft overhead in production.
- Medusa and EAGLE attach lightweight prediction heads to the target itself, avoiding a separate draft model while keeping the lossless verification step.
- Meta's Llama 4 serving uses self-speculative and draft-based decoding to cut decode latency on long generations without changing output quality.
- DeepSeek V4 and other 2026 frontier stacks pair speculative decoding with paged attention and fp8 KV cache as standard decode-latency levers.
- TensorRT-LLM (NVIDIA) implements draft-target speculative decoding and Medusa heads as production reference paths on H100 and B200 hardware.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the corrected token drawn from the residual distribution rather than just resampling the target?
QHow does the acceptance rate determine the wall-clock speedup, and what caps it?
QHow do Medusa and EAGLE differ from a separate draft model?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Claiming the draft must match the target token for token, or that speculation trades a little accuracy for speed. The acceptance rule is exactly lossless; only latency changes.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.