Given draft acceptance rate α and target/draft cost ratio c, when does speculative decoding actually win?

Attempt it

Define the per-round expected accepted tokens and the per-round wall-clock cost for vanilla speculative decoding with draft length K, acceptance rate α, and target/draft cost ratio c. Derive the speedup expression and state the conditions under which speculative decoding LOSES.

Free · 2 AI evals / day

Quick reference

TL;DR

Expected accepted tokens per round is (1 - α^(K+1))/(1 - α); speedup is that over (1 + Kc). Low acceptance or an expensive draft drives it below one.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a slow expert editor and a fast junior writer. The junior drafts the next few words cheaply, then the expert reads all of them in a single glance. The expert keeps the leading words that match what they would have written, and fixes the first word that diverges. If the junior guesses well, the expert confirms several words for the price of one read, so you fly. If the junior guesses badly, the expert keeps almost nothing, yet you still paid for the junior drafting plus the expert read. You also pay a guaranteed correction at the first mismatch. So the trick only pays off when the junior is both cheap and usually right.

Key concepts

Speculative decoding Acceptance rate Draft model

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: derive the truncated geometric expected-tokens sum, build the 1 + Kc cost denominator, write the speedup ratio, then walk both lose regimes and the K optimum.

Real-world examples

Real products, models, and research that use this idea.

vLLM ships speculative decoding with draft models and n-gram proposers, exposing acceptance rate so operators can tune draft length per workload.
Medusa and EAGLE add prediction heads on the target model itself, raising acceptance rate by drafting from the target's own hidden states.
DeepSeek V4 trains a multi-token prediction head used as a built-in draft, pushing acceptance high enough to win on code and structured output.
TensorRT-LLM implements speculative decoding on H100 and B200, where the cheap draft passes overlap the expensive target verify pass.
Llama 4 serving stacks pair a tiny draft model with the flagship target, gating speculation off for high-entropy creative prompts where alpha is low.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the exponent K+1 rather than K in the expected-tokens formula?

Sign in or go Pro to see the approach for this follow-up.

QHow do self-drafting methods like Medusa or EAGLE change the c and alpha tradeoff?

Sign in or go Pro to see the approach for this follow-up.

QHow does large-batch serving interact with speculative decoding's benefit?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Quoting the speedup as a flat K times without the acceptance discount, or ignoring the draft cost in the denominator. Both regimes can push the real speedup below one.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.