Predict whether speculative decoding wins for given α, K, c
Use the speedup formula S = (1 - α^(K+1)) / ((1 - α) * (1 + K * c)). Compute S to two decimal places for these parameters: - α (acceptance rate) = 0.7 - K (draft length) = 4 - c (target/draft cost ratio, i.e. draft cost as a fraction of target cost) = 0.1 Report S as a number with two decimals. Then state whether speculative decoding wins (S > 1) or loses (S < 1) at these settings.
Plug α=0.7, K=4, c=0.1 into the speedup formula and S ≈ 1.98, so speculative decoding nearly doubles throughput here and clearly wins.
Imagine a fast intern who guesses the next four words you are about to write, then you check all four at once instead of writing them yourself. If the intern is usually right, you save tons of time, because one check covers several words. If the intern guesses badly, you keep throwing away the guesses and re-doing the work, so the help is wasted. How often the intern is right is the acceptance rate. The longer the guess and the higher the acceptance rate, the more words you confirm per check. But longer guesses also cost a little extra, since the intern still has to write them. The formula in this question just balances how many words you confirm per check against that small extra cost.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is one of the few inference tricks that buys latency without changing the output distribution. A small draft model proposes K tokens, the large target model verifies all K in one forward pass, and a rejection-sampling correction guarantees the final token stream is distributed exactly as if the target had decoded alone. The payoff is throughput; the risk is that a weak draft wastes the verify passes.
This question hands you a closed-form speedup model and asks for a verdict. The arithmetic is small, but the value is in understanding why the formula has the shape it does, where it is optimistic, and which knob actually moves the result. The dominant knob is the acceptance rate, written α, and the whole point of the sensitivity check is to show that the win-or-lose verdict can flip on α alone.
By the end you should be able to derive the numerator from first principles, explain the denominator's overhead term, plug in the given numbers to land on S ≈ 1.98, and articulate why production teams obsess over measuring α rather than trusting a published figure.
Setting up the speedup model
The speedup formula compares speculative decoding against plain autoregressive decoding, measured as tokens produced per unit of target-model cost. Plain decoding commits exactly one token per target forward pass, so it is the natural baseline of 1.0 against which any speculative scheme is judged.
The target model is the expensive one, so we normalize its cost per forward pass to 1. The draft model is cheaper by a factor c, the draft-to-target cost ratio. Running the draft K times to propose K tokens costs K·c, and one target verify pass costs 1, giving the denominator factor 1 + K·c. The verify pass is parallel: the target scores all K drafted positions in a single batched forward pass, which is why one big verification can confirm many tokens at once.
The numerator counts how many tokens the system commits per verify round on average. A round drafts K tokens, the target verifies them in parallel, accepts a prefix of them, then emits one guaranteed correct token. So the expected committed-token count per round is the expected accepted prefix length plus one. Dividing committed tokens by the per-round cost gives tokens per unit cost, and S is simply that quantity expressed relative to the baseline of one token per target pass. Reading the formula this way, every term has an operational meaning rather than being an opaque algebraic artifact.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with draft models, n-gram proposers, and EAGLE heads, exposing acceptance-rate metrics so operators can tune draft length per workload.
- Medusa adds trained prediction heads to the target model itself, raising acceptance without a separate draft network on Llama-class models.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the exponent K+1 rather than K in the numerator?
Count outcomes. You can accept 0 up to K drafted tokens, and the target always contributes one guaranteed token on top. Summing the geometric series of accept-then-stop events from 0 to K rejections yields the K+1 exponent in the truncated sum.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Raising α to the power K instead of K+1, or forgetting the draft overhead term 1 + K·c in the denominator. Both inflate the predicted speedup.
60 second bullets to scan on the way to the call.
Why the numerator uses the exponent K+1 rather than K
What the denominator overhead term 1 + K times c represents
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.