Same topic, related formats. Practice these next.
In speculative decoding, a small draft model proposes K candidate tokens that a larger target model then verifies. When the target accepts only the first few drafts and rejects the rest, what happens to the target model's KV cache for the rejected positions? Walk through one verification cycle.
Provisionally append K, V for every draft position during verification, then truncate the cache back to the last accepted position when the first rejection happens.
Imagine writing a draft email and showing it to your editor one sentence at a time. You confidently type all five sentences into the shared document, then the editor reads through and approves the first three but flags the fourth. The proper move is to delete sentences four and five entirely so the document holds only what the editor approved, write a corrected sentence in slot four, and start the next round from there. The cache works the same way: anything past the rejection point has to be erased so the next round starts from a clean, agreed-upon prefix.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
6m: cycle structure, parallel target forward over K positions, rejection-sampling acceptance rule, cache truncation on paged vs contiguous layouts, exactness proof intuition, bonus-token rule, and when acceptance rates make the technique net-negative.
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Leaving K, V for rejected positions in the cache because 'memory is cheap'. The next step's attention would then dot against tokens that no longer exist in the sequence, corrupting every future generation.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.