In speculative decoding, a small draft model proposes K candidate tokens that a larger target model then verifies. When the target accepts only the first few drafts and rejects the rest, what happens to the target model's KV cache for the rejected positions? Walk through one verification cycle.
Provisionally append K, V for every draft position during verification, then truncate the cache back to the last accepted position when the first rejection happens.
Imagine writing a draft email and showing it to your editor one sentence at a time. You confidently type all five sentences into the shared document, then the editor reads through and approves the first three but flags the fourth. The proper move is to delete sentences four and five entirely so the document holds only what the editor approved, write a corrected sentence in slot four, and start the next round from there. The cache works the same way: anything past the rejection point has to be erased so the next round starts from a clean, agreed-upon prefix.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Speculative decoding is the most widely deployed inference-time speedup for autoregressive LLMs in 2026, and the KV-cache bookkeeping is the part that distinguishes a correct implementation from a subtly broken one. The cache must hold exactly the K and V for the committed sequence at the end of every cycle, no leftover entries from rejected drafts, no missing entries from accepted ones.
The technique itself is simple. A small draft model proposes K tokens. A large target model verifies them in a single parallel forward pass. Accepted tokens commit, rejected positions are rolled back, and one bonus token is sampled to guarantee forward progress. The output distribution is provably identical to plain target decoding under a modified rejection-sampling rule.
This deep dive walks through one full cycle, the rollback mechanics on different cache layouts, the exactness proof intuition, and the operational considerations that determine whether speculative decoding pays off.
Anatomy of one verification cycle
The cycle starts with the draft model running K autoregressive steps to produce K candidate tokens. The draft uses its own (small) KV cache for these steps.
The target's parallel forward
The target model then takes those K candidates and runs ONE forward pass over all K positions in parallel. This is the central trick: instead of K serial decode steps each costing one full target forward, you pay one target forward for K positions. The parallel pass computes K, V, and logits at every candidate position.
K and V land in the target's cache as a batched append. Logits are used by the acceptance test, which runs left to right. At each position the test compares the target distribution p_target to the draft distribution p_draft and accepts with probability min(1, p_target(x) / p_draft(x)). The first rejection ends the run.
Two outcomes
The cycle has two paths from here:
- All K accepted: commit all K tokens, plus one bonus token sampled from p_target at position K+1 (the target already has the logits because the parallel forward extends one position past the drafts). Cache holds K+1 new entries.
- First rejection at position j: commit positions 1 through j-1, resample position j from the residual distribution max(0, p_target minus p_draft) renormalized, and roll back the cache to remove entries for positions j+1 through K.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships speculative decoding with a draft model and rolls back PagedAttention pages on rejection, used in production at OpenAI-style serving providers in 2026.
- EAGLE-2 and EAGLE-3 train a tiny draft head on top of the target's hidden states, eliminating a separate draft model and improving acceptance rates above 80 percent on aligned text.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat changes if the draft and target use different tokenizers?
Tokenizer mismatch breaks the per-position acceptance test because the candidate sequences live in different token spaces. You either re-tokenize the draft output into target tokens (lossy and slow) or constrain the draft to the target's tokenizer. Most production deployments train a draft head that shares the target's vocab to avoid the problem entirely.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Leaving K, V for rejected positions in the cache because 'memory is cheap'. The next step's attention would then dot against tokens that no longer exist in the sequence, corrupting every future generation.
60 second bullets to scan on the way to the call.
Where K and V for draft positions live during the verification pass
How the parallel forward over K positions differs from K serial steps
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.