Put the steps of one round of vanilla speculative decoding in the correct order
- 1Draft model autoregressively proposes K candidate tokens t_1, ..., t_K
- 2Accepted tokens (plus the resampled correction) are appended; draft KV cache is rewound to the new tip and the round repeats
- 3For each position i = 1..K, the verifier checks whether the target's distribution accepts t_i under the speculative sampling rule
- 4Target model runs ONE forward pass over the K proposed tokens in parallel, computing logits at every position
- 5On the first rejection (or after all K accepted), the target's own distribution at that position is sampled to produce a corrected token
A cheap draft proposes K tokens, the target verifies all K in one parallel pass, accepts the matching prefix, resamples the first reject, then repeats. The acceptance rule keeps the target's distribution exact.
Imagine a slow but careful editor and a fast but sloppy assistant. The assistant guesses the next five words quickly. Instead of writing each word himself, the editor reads all five guesses at once in a single glance and checks them in order. He keeps the guesses that match what he would have written, and stops at the first one he disagrees with. There he writes his own word, throws away the rest of the guesses, and lets the assistant guess again from that point. Because the editor only ever keeps words he approves, the final text reads exactly as if he had written every word himself, just much faster when the assistant guesses well.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is the rare inference optimization that buys real latency reduction without sacrificing one bit of output quality. That combination is exactly why interviewers like it. A candidate who can put the round in the correct order AND argue why the output is distributionally identical to the target has understood both the systems angle and the probability angle.
The core bottleneck it attacks is that autoregressive decode is sequential and memory bandwidth bound. Each token requires a full forward pass that streams the weights and KV cache from HBM, and the arithmetic units sit largely idle. Speculative decoding fills that idle compute by letting a cheap draft model guess several tokens ahead, then verifying all of them in a single target pass that costs almost the same as decoding one token.
This deep dive walks through the five ordered steps of one round, explains why each step must sit where it does, and then proves the part that trips up most candidates: why accepting the matching prefix and resampling the first rejection yields a stream that is exactly distributed as the target model decoding alone. By the end you should be able to defend both the ordering and the correctness guarantee under follow-up pressure, and to reason about when speculation pays off versus when it wastes work.
Keep one framing in mind throughout. The draft and the target play asymmetric roles. The draft is a throughput device that proposes cheaply and is allowed to be wrong. The target is the source of truth that both verifies and, when needed, corrects. Every step in the round either generates a cheap proposal, evaluates it against the source of truth, or reconciles the two. Once you internalize that split, the ordering and the math both follow naturally.
Step 1: the draft proposes K tokens
The round opens with the draft model, a small cheap model, decoding K tokens autoregressively from the current sequence tip. This is ordinary greedy or sampled decoding, just on a model that might be 10 to 50 times smaller than the target.
The draft pays the sequential cost here: K small forward passes, one per proposed token. Because the draft is tiny, those K passes are cheap relative to a single target pass. The draft also records the probability it assigned to each token it proposed, because the verification step needs that draft probability.
The choice of K is a tuning knob. A larger K means more speculation per expensive pass when guesses land, but more wasted draft work and more discarded tokens when an early rejection happens. Typical K sits between 4 and 8 in 2026 serving stacks.
There is a subtlety in what 'proposes' means. The draft does not have to greedily pick its top token. It samples from its own distribution, and it is the draft probability of the token it actually emitted that gets carried forward into verification. This matters because the acceptance rule compares the target probability of that specific token against the draft probability of that same token. If the draft were forced to argmax, the proposed sequence would be more predictable but the acceptance math would still hold, since the rule is defined per proposed token regardless of how the draft chose it.
The draft also reuses its own KV cache across the K steps, so proposing K tokens is K cheap incremental decodes, not K full prefills. That keeps the proposal phase genuinely cheap relative to the one target pass that follows.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with both draft model and n-gram proposers, configurable per request in 2026 deployments.
- Medusa attaches extra decoding heads to the target itself, removing the separate draft model while keeping the verify and accept loop.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does accepting the matching prefix and resampling the first rejection preserve the exact target distribution?
Walk through the speculative sampling identity. Accept with probability that is the minimum of one and the target over draft ratio, and on rejection sample from the normalized positive part of target minus draft. Show the resulting per-position marginal equals the target for any draft.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking the target runs once per proposed token, or that accepting only the matching prefix changes the output distribution. The verification step is exact, not approximate.
60 second bullets to scan on the way to the call.
The five-step order of one speculative decoding round
Why the single target pass follows the full proposal of K tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.