Spot the errors in this claim about speculative decoding

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Speculative decoding only wins when the draft is accepted often and the target is memory-bound. At large batch the GPU is compute-bound, so verifying drafts steals FLOPs and can slow it.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a slow expert editor who must approve every word of a book. To save time, a fast junior writer drafts the next few words and the editor checks them all at once instead of writing each word alone. If the junior guesses well, the editor approves a whole batch in one glance and the book finishes faster. But if the junior keeps guessing wrong, the editor throws the draft away and writes it himself anyway, so you paid for the junior for nothing. And if the editor is already swamped handling a hundred books at once, every minute spent checking guesses is stolen from real writing. The trick only helps when the editor is idle and the junior is usually right.

Key concepts

Speculative decoding Acceptance rate Memory-bound vs compute-bound Batching

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: mechanism (draft + verify) + the two knobs alpha and c + why memory-bound small batch wins and compute-bound large batch loses + map the three prompt errors.

Real-world examples

Real products, models, and research that use this idea.

vLLM ships speculative decoding with draft models and n-gram proposers, and its docs warn the gain shrinks as batch size rises toward compute-bound serving.
Medusa and EAGLE add lightweight draft heads to a target like Llama 4, raising acceptance on structured text without a separate draft model.
TensorRT-LLM (NVIDIA) exposes speculative decoding on H100 and B200 but recommends it mainly for low-concurrency latency-sensitive deployments.
Anthropic and OpenAI use speculative methods to cut single-stream latency on Claude Opus 4.7 and GPT-5.5 class chat, where batch per replica stays modest.
DeepSeek V4 pairs multi-token prediction heads with speculative verification to accelerate single-stream decode without a separate draft model.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the speculative speedup shrink as batch size grows?

Sign in or go Pro to see the approach for this follow-up.

QHow does the acceptance rate alpha interact with the draft window length K?

Sign in or go Pro to see the approach for this follow-up.

QHow do Medusa and EAGLE change the cost economics versus a separate draft model?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating speculative decoding as a universal speedup. It only helps in the memory-bound, low-batch regime with high acceptance. At large batch the target is compute-bound and verifying drafts steals FLOPs.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.