Speculative Decoding
Also known as: Assisted decoding
Use a small draft model to propose multiple tokens, then have the big model verify them in parallel.
An inference acceleration technique where a small 'draft' model proposes several tokens which the large 'target' model then verifies in parallel. Accepts the prefix that matches, rejects the rest. Roughly doubles tokens/sec with no quality change.
In practice
Shipped in vLLM, TensorRT-LLM, and llama.cpp. Senior interviews probe how the parallel verify works and why it's exact (not approximate).
Related topics
Questions that mention this term
- DeepSeek trains models to predict K tokens per step. Pick how that stays causal.
- Match EAGLE-3 and Medusa-2 to drafting mechanism, integration cost and acceptance behavior
- Why does tree based speculative decoding outperform sequential draft chains at the same target pass cost?
- Describe why structured output mode quietly raises the per call token bill
- How does vanilla speculative…
- Pick the serving changes TensorRT-LLM 0.18 (early 2026) actually shipped
Related terms
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.