Spot the errors in this claim about speculative decoding

**Speculative decoding** uses a small, fast draft model to guess several next tokens, then lets the big target model check all of them in a single forward pass. If the guesses are good, you produce many tokens for the price of one target step. If they are bad, you fall back and only keep what the target would have produced anyway. The reason this can help at all is that single-stream decoding is memory-bound. The GPU spends most of its time reading model weights and the KV cache from memory, not doing math. So verifying several tokens at once costs almost the same as verifying one, and the extra accepted tokens are nearly free. The claim in the prompt gets this exactly backwards. It says the speedup is largest at maximum throughput with hundreds of concurrent requests. At that point the GPU is already saturated with arithmetic, so the wasted verification work has nowhere to hide and can slow you down. It also calls rejection free and the technique always a win. Neither is true. On creative writing the draft agrees rarely, acceptance drops, and most of the drafting and verification effort is thrown away.

**Speculative decoding** runs a cheap draft model to propose several tokens, then verifies them in one target forward pass via rejection sampling. The output is provably identical to plain target sampling. The win is a latency win, and it is conditional. The speedup depends on two knobs. One is the acceptance rate alpha, how often the target accepts a drafted token. The other is the cost ratio c, the draft step cost over the target step cost. You profit only when accepted tokens per cycle outrun the added draft and verify overhead. Speedup is: $$S \approx \frac{1 - \alpha^{K+1}}{(1 - \alpha)(1 + c K)}$$ The deeper condition is the hardware regime. Single-stream decoding is memory-bandwidth-bound. Verifying several tokens then costs about the same as verifying one. That free headroom is what speculative decoding spends. At large batch the target is already compute-bound. There is no idle arithmetic to reclaim. Verifying K drafts now multiplies the target FLOPs per step, and rejected drafts waste real compute. So the technique can become a net loss in the high-throughput regime. The prompt's three errors all stem from ignoring alpha and the batch-size regime.