How does vanilla speculative decoding work and why is verification a single forward pass?

Attempt it

Walk through one round of vanilla speculative decoding. Be specific about (a) why the target's verification is a single forward pass even though K tokens are being checked, and (b) what guarantees the final distribution is identical to running the target alone.

Free · 2 AI evals / day

Quick reference

TL;DR

A cheap draft proposes K tokens; the target verifies all K in one parallel forward pass; a rejection-sampling rule makes the output distribution exactly match the target alone.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a fast but sloppy assistant who guesses the next several words of your sentence while you, the careful editor, are still thinking. The editor reads all the guesses at once in a single glance instead of word by word. Wherever a guess matches what the editor would have written, the editor keeps it for free. At the first guess the editor disagrees with, the editor crosses it out, writes the correct word, and throws away every guess after it. Because the editor still makes the final call on each word using a fair coin-flip rule, the finished sentence reads exactly as if the editor wrote every word alone. The only thing the assistant changed is speed, never the final wording.

Key concepts

Speculative decoding Draft and target models Modified rejection sampling Memory-bound decode

Quick scan · 2–4 min

Everything important, quickly.

4 min: draft proposes K tokens + target single-pass verification via causal mask + acceptance ratio rule + residual resample on rejection + the lossless equivalence + memory-bound speedup intuition.

Code example

python

# One round of vanilla speculative decoding (illustrative)
draft_tokens, p_draft = draft.propose(prefix, k=K)        # K cheap autoregressive steps
p_target = target.forward(prefix + draft_tokens)          # ONE pass, logits at all K positions

accepted = []
for i, t in enumerate(draft_tokens):
    r = random.random()
    if r < min(1.0, p_target[i][t] / p_draft[i][t]):
        accepted.append(t)                                # accept draft token
    else:
        residual = relu(p_target[i] - p_draft[i])         # max(0, p_t - p_d)
        accepted.append(sample(residual / residual.sum()))
        break                                             # discard positions after j
else:
    accepted.append(sample(p_target[K]))                  # bonus token when all K accepted

Side-by-side

Aspect	Standard autoregressive decode	Speculative decoding
Target passes per token block	One target pass per single token	One target pass verifies up to K tokens
Output distribution	Target distribution	Exactly the target distribution (lossless)
Extra compute	None	K cheap draft passes per round
Speedup source	None	More tokens advanced per memory-bound weight read
Failure mode	Latency floor at one token per pass	Low acceptance wastes draft work and verification

Real-world examples

Real products, models, and research that use this idea.

vLLM ships speculative decoding with draft model, n-gram, and EAGLE-style methods, letting operators trade acceptance rate against draft overhead in production.
Medusa and EAGLE attach lightweight prediction heads to the target itself, avoiding a separate draft model while keeping the lossless verification step.
Meta's Llama 4 serving uses self-speculative and draft-based decoding to cut decode latency on long generations without changing output quality.
DeepSeek V4 and other 2026 frontier stacks pair speculative decoding with paged attention and fp8 KV cache as standard decode-latency levers.
TensorRT-LLM (NVIDIA) implements draft-target speculative decoding and Medusa heads as production reference paths on H100 and B200 hardware.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the corrected token drawn from the residual distribution rather than just resampling the target?

Sign in or go Pro to see the approach for this follow-up.

QHow does the acceptance rate determine the wall-clock speedup, and what caps it?

Sign in or go Pro to see the approach for this follow-up.

QHow do Medusa and EAGLE differ from a separate draft model?

Sign in or go Pro to see the approach for this follow-up.

QWhat breaks the single-pass verification if the draft proposes a tree of candidates instead of one chain?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.