Match each advanced speculative decoding variant to its defining mechanism
Drag each answer to line up with its matching prompt
Vanilla speculative decoding
Jacobi iteration over an n-gram pool: self-speculation with no draft model and no training
Medusa
Multiple trained decoding heads on the target model emit K candidates in parallel, no separate draft model
Eagle / EAGLE-2
All four trade extra parallel compute per round for fewer sequential decode steps
Lookahead decoding
Predicts the target's future HIDDEN states (not just tokens) to drive higher acceptance
Common payoff axis
Separate small draft model proposes K tokens autoregressively; target verifies in one pass
All four variants cut sequential decode steps by guessing several tokens at once: Medusa uses extra heads, EAGLE predicts hidden states, lookahead runs Jacobi over n-grams, vanilla uses a draft model.
Imagine dictating a letter to a typist who can only write one word at a time, then waits for you to say the next. Slow. Speculative decoding adds an apprentice who guesses the next few words ahead, and you just check the guesses in one glance, keeping the correct ones. The variants differ in who guesses. A separate junior typist is vanilla speculative decoding. Extra hands on your own typist that scribble several guesses at once is Medusa. A typist who anticipates your train of thought, not just your words, is EAGLE. And lookahead is the typist replaying common phrases they have already typed and checking which ones fit. Either way, you verify many words per glance instead of one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is the dominant family of latency optimizations for autoregressive LLM serving in 2026, and the advanced variants (Medusa, EAGLE, and lookahead decoding) are a favorite senior-interview probe precisely because they look similar but differ on a single sharp axis: who produces the draft, and at what cost.
The shared premise is a hardware fact. During decode the GPU spends most of its time streaming model weights and the KV cache from HBM, not doing arithmetic. A forward pass that scores K candidate tokens costs almost the same wall-clock time as one that scores a single token, because both are bottlenecked on memory bandwidth, not flops. That slack is free money, and every technique here is a different scheme for spending it.
The questions in this match expose a common confusion: people lump all four together as 'small model guesses, big model checks.' That is only the vanilla story. The frontier variants moved the drafter inside the target model or removed it entirely, and the interview value is in articulating exactly that progression.
This deep dive separates the four. It explains the verification mechanism that makes speculation lossless, then walks each variant's drafter, then closes on the operating regime where speculation helps and where it quietly hurts. By the end you should be able to match each name to its mechanism instantly and defend the matching under follow-up pressure.
The bottleneck that makes speculation possible
Autoregressive generation is sequential by construction. Token T+1 depends on token T, so the model cannot start the next step until the current one finishes. Each step loads the full weight set and the growing KV cache from HBM to emit exactly one token.
That makes decode memory-bandwidth-bound at low batch sizes. The arithmetic units are mostly idle while data streams in. Crucially, a single forward pass can score many positions at once almost for free, because the expensive part, moving weights, is already paid.
Speculation turns that idle compute into saved steps. You produce K candidate future tokens cheaply, run one target forward pass over all K, and check which prefix the target would itself have generated. Accepted tokens advance the sequence; the first rejection truncates the rest. The number of slow sequential steps drops by the average accepted length per round.
The expected speedup is governed by how many tokens survive per verification round:
The numerator is the average accepted length, and the denominator folds in the relative cost of producing the draft. This single ratio explains every design choice that follows. Vanilla buys a higher numerator with a real draft model but pays draft cost in the denominator. Medusa and lookahead drive the denominator toward zero by drafting almost for free. EAGLE attacks the numerator by making each drafted token far more likely to be accepted.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Variant | Drafter | Needs training | Needs second model |
|---|---|---|---|
| Vanilla speculative | Separate small model | No (reuse a small model) | Yes |
| Medusa | Extra heads on target | Yes (heads) | No |
| EAGLE / EAGLE-2 | Hidden-state predictor | Yes (feature head) | No |
| Lookahead | Jacobi n-gram pool | No | No |
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with draft-model, n-gram, and EAGLE-style proposers selectable per deployment in 2026.
- Medusa heads are bundled into TensorRT-LLM and were used to accelerate Vicuna and later Llama-family chat serving.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does exact speculative decoding leave the target's output distribution unchanged?
Walk through the modified rejection sampling step. The target accepts a drafted token with a probability tied to the ratio of target to draft probabilities, and resamples from an adjusted residual distribution on rejection. The math is constructed so the marginal equals pure target sampling.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming all four use a separate draft model. Only vanilla does. Medusa and EAGLE bolt onto the target, and lookahead has no draft model and no training at all.
60 second bullets to scan on the way to the call.
Why decode is memory bound and how that creates room for speculation
Which variants draft from the target versus a separate model
Primary sources. Browse if you want the original framing.
- Leviathan et al., Fast Inference from Transformers via Speculative Decoding
- Cai et al., Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads
- Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Fu et al., Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
Same topic, related formats. Practice these next.