Speculative Decoding
Also known as: Assisted decoding
Use a small draft model to propose multiple tokens, then have the big model verify them in parallel.
An inference acceleration technique where a small 'draft' model proposes several tokens which the large 'target' model then verifies in parallel. Accepts the prefix that matches, rejects the rest. Roughly doubles tokens/sec with no quality change.
In practice
Shipped in vLLM, TensorRT-LLM, and llama.cpp. Senior interviews probe how the parallel verify works and why it's exact (not approximate).
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.