Given draft acceptance rate α and target/draft cost ratio c, when does speculative decoding actually win?

**Speculative decoding** uses a small fast draft model to propose the next few tokens, then the big target model verifies all of them in one parallel pass. Tokens that match what the target would have sampled are accepted for free, so you can emit several tokens per expensive target pass instead of one. The catch is that acceptance is probabilistic. Model each proposed token as an independent coin flip with success probability alpha, the acceptance rate. You accept a run of leading tokens until the first mismatch, then the target supplies one corrected token at that point. The expected number of tokens you emit per round follows a geometric pattern in alpha, capped at the draft length K. More draft tokens give a longer potential run, but each extra one is less likely to survive. The cost side has two parts: one full target pass to verify, plus K cheap draft passes. We measure draft cost relative to a target pass with a ratio c. The scheme only wins when the tokens gained outrun this combined cost, which needs the draft both accurate and genuinely cheap.

**Speculative decoding** drafts K tokens with a cheap model, then verifies them in one target pass. Model each draft token as a Bernoulli trial with acceptance probability alpha. You accept a leading run, then rejection sampling always emits one corrected token at the first mismatch. Summing that truncated geometric series gives the expected accepted tokens per round: $$\mathbb{E}[\text{tokens}] = \frac{1 - \alpha^{K+1}}{1 - \alpha}$$ The per-round cost is one target pass plus K draft passes. Measure draft cost as a fraction c of one target pass. Cost is roughly 1 + Kc target-units. The speedup is the tokens gained divided by that cost: $$S \approx \frac{1 - \alpha^{K+1}}{(1 - \alpha)\,(1 + Kc)}$$ Speculative decoding loses when S falls below one. Two regimes cause this. Low alpha kills the numerator, since high-entropy generation rarely matches a small draft. High c inflates the denominator, since a draft that is not much cheaper than the target wastes the parallel pass. For fixed alpha there is also an optimal K. Too small leaves throughput unused. Too large pays draft cost for tokens that get rejected.