Which regimes can make speculative decoding fail to speed up, or actively slow down, a serving stack?

**Speculative decoding** uses a small fast draft model to propose several tokens at once, then the big target model verifies all of them in one forward pass. Accepted tokens are kept and the first rejection is corrected. When most proposals are accepted, you generate several tokens for the price of one target pass. The trick exploits a specific fact: at small batch sizes, decode is memory bound. The GPU spends most of its time reading model weights and the KV cache from memory, while the arithmetic units sit largely idle. Verifying several draft tokens in parallel uses that idle compute almost for free. That is exactly why it can fail. If the draft and target rarely agree, most proposals get rejected, and you paid for draft passes that bought nothing. If the draft model is itself slow or large, each proposal cycle costs too much. And if the batch is already large, the GPU is compute bound, so there is no idle compute to spend on verification. The mental model: speculative decoding converts spare FLOPs into lower latency. No spare FLOPs, low agreement, or an expensive draft each break the trade, and the overhead can make decoding slower than plain generation.

**Speculative decoding** trades spare compute for latency. A draft model proposes K tokens and the target verifies them in one parallel pass. It wins only when decode is memory bound, so verification rides on otherwise idle FLOPs. The expected speedup depends on two numbers. Acceptance rate is how often a draft token survives verification. Cost ratio is the draft pass cost divided by the target pass cost. High acceptance and a cheap draft mean a big win. Four regimes break it. First, high-entropy outputs like creative writing drive acceptance down, because draft and target disagree on flat distributions. Second, a draft that is too large or slow pushes the cost ratio toward one, so the extra draft passes nearly equal a target pass. Third, domain mismatch between draft and target collapses acceptance even within one model family. Fourth, and most subtle, a large decode batch is already compute bound. That last regime matters most. At large batch the target saturates the GPU, so there is no idle compute to convert. Verification then adds raw work and can lower total throughput. The clean framing: speculative decoding is a latency optimization for the low-batch memory-bound regime, not a throughput optimization.