Given draft acceptance rate α and target/draft cost ratio c, when does speculative decoding actually win?
Define the per round expected accepted tokens and the per round wall clock cost for vanilla speculative decoding with draft length K, acceptance rate α, and target/draft cost ratio c. Derive the speedup expression and state the conditions under which speculative decoding LOSES.
Expected accepted tokens per round is (1 - α^(K+1))/(1 - α); speedup is that over (1 + Kc). Low acceptance or an expensive draft drives it below one.
Imagine a slow expert editor and a fast junior writer. The junior drafts the next few words cheaply, then the expert reads all of them in a single glance. The expert keeps the leading words that match what they would have written, and fixes the first word that diverges. If the junior guesses well, the expert confirms several words for the price of one read, so you fly. If the junior guesses badly, the expert keeps almost nothing, yet you still paid for the junior drafting plus the expert read. You also pay a guaranteed correction at the first mismatch. So the trick only pays off when the junior is both cheap and usually right.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is one of the few inference techniques that delivers a wall-clock speedup with zero change to the output distribution. That guarantee is what makes it a favorite hard interview question. The candidate cannot hand-wave about approximate quality tradeoffs, because there are none. The entire conversation is about throughput arithmetic, and the arithmetic has two knobs that decide whether the technique wins or quietly loses.
The core idea is asymmetric work. A small draft model proposes the next several tokens cheaply and sequentially. The large target model then verifies all of those proposals in a single parallel forward pass, the same way it would process a prompt during prefill. If the draft guessed a token the target also would have sampled, that token is accepted for free, because the target pass that confirmed it would have cost the same whether it verified one token or several.
This deep dive derives the two governing quantities from scratch: the expected number of tokens emitted per verification round, and the wall-clock cost of that round. Dividing one by the other gives the speedup. We then read off exactly when the speedup drops below one, why draft length has an optimum, and which production systems chase higher acceptance to widen the winning regime.
The acceptance process as a truncated geometric chain
Fix a draft length K, the number of tokens the small model proposes before each target verification. Verification walks the proposed tokens left to right. Standard analysis models each proposed token as accepted with probability alpha, the acceptance rate, independently of the others.
Acceptance stops at the first rejected token. So the number of accepted draft tokens is a geometric random variable truncated at K: you accept the first token with probability alpha, the first two with probability alpha squared, and so on. Crucially, the rejection sampling correction step always emits one valid token at the position of the first mismatch, sampled from the adjusted target distribution. If all K drafts are accepted, the target's own next-token prediction supplies the bonus token.
This means every round emits at least one token and at most K+1 tokens. The floor of one is what keeps the scheme from ever being worse than plain decoding in tokens emitted. The plus-one bonus is the detail candidates most often drop.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with draft models and n-gram proposers, exposing acceptance rate so operators can tune draft length per workload.
- Medusa and EAGLE add prediction heads on the target model itself, raising acceptance rate by drafting from the target's own hidden states.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the exponent K+1 rather than K in the expected-tokens formula?
Count both the accepted draft prefix and the one guaranteed corrected token at the first rejection. The geometric sum runs over K+1 emission slots, so the truncation appears as alpha to the K+1.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quoting the speedup as a flat K times without the acceptance discount, or ignoring the draft cost in the denominator. Both regimes can push the real speedup below one.
60 second bullets to scan on the way to the call.
The truncated geometric sum for expected accepted tokens and its K+1 exponent
Why a corrected token is always emitted at the first rejection
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.