Same topic, related formats. Practice these next.
Speculative decoding loses when acceptance is low, the draft is too costly, or the batch is already compute-bound: it converts spare FLOPs into latency, so with no spare FLOPs there is nothing to convert.
Imagine a slow expert chef plating one dish at a time. You hire a fast apprentice to guess the next few dishes ahead, and the expert just glances to confirm them in one go. If the apprentice guesses well, the expert approves a whole batch at once and you finish faster. But if the apprentice keeps guessing wrong, the expert redoes everything, so the help was wasted. And if the kitchen is already slammed cooking for a hundred tables at once, the expert has no spare moment to check guesses, so the apprentice only adds to the pile. The trick helps only when guesses are good, the apprentice is cheap, and the kitchen has idle hands to spare.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: the spare-FLOPs-for-latency trade + acceptance rate and cost ratio + the four failure regimes + why large batch is the killer + the FP8 KV distractor.
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating speculative decoding as a free win. It helps only when decode is memory bound with spare compute; at large batch the GPU is already compute saturated and verification just adds work.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.