Does speculative decoding win for given α, K, c
Use the speedup formula S = (1 - α^(K+1)) / ((1 - α) * (1 + K * c)). Compute S to two decimal places for these parameters: - α (acceptance rate) = 0.7 - K (draft length) = 4 - c (draft/target cost ratio, i.e. draft cost as a fraction of target cost) = 0.1 Report S as a number with two decimals. Then state whether speculative decoding wins (S > 1) or loses (S < 1) at these settings.
Plug α=0.7, K=4, c=0.1 into the speedup formula and S ≈ 1.98, so speculative decoding nearly doubles throughput here and clearly wins.
Imagine a fast intern who guesses the next four words you are about to write, then you check all four at once instead of writing them yourself. If the intern is usually right, you save a lot of time, because one check covers several words. If the intern guesses badly, you keep throwing away the guesses and re-doing the work, so the help is wasted. How often the intern is right is the acceptance rate. The longer the guess and the higher the acceptance rate, the more words you confirm per check. But longer guesses also cost a little extra, since the intern still has to write them. The formula balances how many words you confirm per check against that small extra cost.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is one of the few inference methods that cuts latency without changing the output distribution. A small draft model proposes K tokens, the large target model verifies all K in one forward pass, and a rejection-sampling correction guarantees the final token stream is distributed exactly as if the target had decoded alone. The payoff is throughput; the risk is that a weak draft wastes the verify passes.
A closed-form speedup model gives a win-or-lose verdict. The dominant input is the acceptance rate, written α, and the sensitivity check shows that the verdict can flip on α alone.
Setting up the speedup model
The speedup formula compares speculative decoding against plain autoregressive decoding, measured as tokens produced per unit of target-model cost. Plain decoding commits exactly one token per target forward pass, so it is the natural baseline of 1.0 against which any speculative scheme is judged.
The target model is the expensive one, so its cost per forward pass is normalized to 1. The draft model is cheaper by a factor c, the draft-to-target cost ratio. Running the draft K times to propose K tokens costs K·c, and one target verify pass costs 1, giving the denominator factor 1 + K·c. The verify pass is parallel: the target scores all K drafted positions in a single batched forward pass, which is why one big verification can confirm many tokens at once.
The term (1 − α^(K+1)) / (1 − α) counts how many tokens the system commits per verify round on average. A round drafts K tokens, the target verifies them in parallel, accepts a prefix of them, then emits one guaranteed correct token. So the expected committed-token count per round is the expected accepted prefix length plus one. Dividing committed tokens by the per-round cost gives tokens per unit cost, and S is that quantity expressed relative to the baseline of one token per target pass. Read this way, every term has an operational meaning.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with draft models, n-gram proposers, and EAGLE heads, exposing acceptance-rate metrics so operators can tune draft length per workload.
- Medusa adds trained prediction heads to the target model itself, raising acceptance without a separate draft network on Llama-class models.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the exponent K+1 rather than K in the numerator?
Count outcomes. You can accept 0 up to K drafted tokens, and the target always contributes one guaranteed token on top. Expected committed tokens are the sum of alpha^i for i = 0 to K, which has K+1 terms, so its closed form has the exponent K+1.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Raising α to the power K instead of K+1, or forgetting the draft overhead term 1 + K·c in the denominator. The first understates the predicted speedup and the second inflates it.
60 second bullets to scan on the way to the call.
Why the numerator uses the exponent K+1 rather than K
What the denominator overhead term 1 + K times c represents
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.