Speculative decoding promises faster tokens — when does it actually deliver?

Attempt it

Describe how speculative decoding lowers decode latency and the condition under which it stops paying off.

Free · 2 AI evals / day

Quick reference

TL;DR

Speculative decoding has a cheap draft model guess several tokens that the big model verifies in one pass — it only wins when acceptance is high.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a slow expert proofreading a document. Instead of writing one word at a time, a fast intern jots down the next few words they expect, and the expert glances at the whole guess at once — keeping the part that's right and fixing the first wrong word. When the intern guesses well, the expert finishes way faster because checking a sentence is barely slower than checking one word. When the intern guesses badly, the expert ends up redoing most of it, and the trick saves almost nothing.

Key concepts

LLM System Design Inference Optimization

Quick scan · 2–4 min

Everything important, quickly.

Spend 6-8 minutes connecting the draft-verify mechanism to decode's memory-bandwidth limit, then derive when acceptance rate makes it net positive.

Side-by-side

Regime	Acceptance	Net effect
Predictable / in-distribution text	High (e.g. 70-90%)	Several tokens per target step; large latency cut
Hard / out of distribution text	Low (e.g. 20-30%)	Draft + verify cost barely amortized; little gain
Heavy continuous batching	Any	GPU already compute-bound; spare capacity shrinks, win fades

Real-world examples

Real products, models, and research that use this idea.

vLLM ships speculative decoding with draft-model, n-gram, and EAGLE/Medusa-style options for production serving.
Anthropic and OpenAI use speculative-style techniques to reduce per-token latency on frontier chat models.
NVIDIA TensorRT-LLM supports draft-target speculative decoding for low-latency GPU inference.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the rejection-sampling acceptance rule preserve the target distribution exactly?

Sign in or go Pro to see the approach for this follow-up.

QHow do Medusa and EAGLE remove the separate draft model?

Sign in or go Pro to see the approach for this follow-up.

QHow would you tune the speculation length k for a given workload?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.