Prefix-cache hit lands on a request: does speculative decoding still help that request?

Short answer·Medium·4.0 · 0·~3 min·Relevant atNVIDIA

Attempt it

Explain how a prefix-cache hit changes the latency profile of a request, and whether that change makes speculative decoding more, less, or equally valuable for the same request. Argue from where each technique actually saves time.

Free · 2 AI evals / day

**The two techniques target different phases of the request.** A request to an LLM has two phases: prefill (process the entire prompt in one parallel pass) and decode (generate the response one token at a time). They have different bottlenecks and respond to different optimizations. **Prefix cache** saves prefill work. When the serving system has already processed the same prompt prefix for an earlier request, it has already computed and stored the KV-cache entries for that span. A cache hit means the prefill step reuses those entries and only computes the tail beyond the cached region. This can drop prefill from hundreds of milliseconds to a few milliseconds on long-context workloads like RAG over a shared knowledge base. **Speculative decoding** saves decode work. A small draft model (or extra heads on the target model) proposes several candidate next tokens. The target model then verifies those candidates in a single forward pass and accepts the ones that match its own distribution. On a successful acceptance run, you got multiple tokens for one HBM sweep of the target's weights, which is the dominant cost in memory-bandwidth-bound decode. So the cache touches phase A and speculation touches phase B. A cache hit changes how long prefill takes; it does not change how long each decode step takes or what acceptance rate the draft model achieves. Speculation gives the same relative win whether prefill was cached or not. In practice the two compose well. Long-prefix decode-heavy workloads (RAG, long system prompts plus medium completions) get both. The cache cuts the prefill tax; speculation cuts the decode tax; total request latency drops on both fronts.

**They attack different halves of the request, so they stack.** Prefix cache reduces prefill by reusing KV entries for a shared prompt prefix. Decode is unchanged: the model still generates one token at a time, still reads the entire KV cache from HBM each step, and is still memory-bandwidth bound exactly as before. The cache hit shrinks the front of the request, not the per-token step. Speculative decoding lives entirely inside the decode loop. A draft model proposes several candidate tokens; the target verifies them in one forward pass; accepted candidates yield multiple tokens per HBM sweep of the target's weights. The speedup is a function of decode-step cost times acceptance rate, both intrinsic to the decode loop and the target/draft pair. Neither quantity changes when prefill happens to be cached. **Conclusion: speculation's relative win on a cache-hit request equals its relative win on a cache-miss request.** The absolute latency improvement is in the decode segment, which is the same segment in both cases. Total request time differs only in the prefill prefix, which the cache hit shortened. The two compose cleanly in long-prefix decode-heavy workloads. RAG over a shared knowledge base has long, repeating prefixes (the cache wins) and meaningful response lengths (speculation wins). A code-assistant with long system prompts is the same shape. The right operational posture: enable both, tune acceptance rate independently of cache hit rate, monitor each savings metric separately on its own phase.

Quick reference

TL;DR

Prefix cache saves prefill, speculation saves decode. They are orthogonal, and speculation's relative win on a cache-hit request is unchanged. The two stack and you want both for long-prefix RAG.

Memory aid

Easy to grasp

Think of a long bus trip in two parts: a slow scenic drive to the city (prefill), then a fast highway run home (decode). A prefix cache is like a teleporter that skips the scenic drive when you have made the same trip before; it has nothing to do with how fast the highway part goes. Speculative decoding is a faster engine for the highway run; it has nothing to do with whether you teleported there or drove. If you teleport in and then drive home, the faster engine still gives you the same time savings on the highway as it would after a normal scenic drive. The two improvements affect different legs of the same trip, so they stack rather than overlap.

Key concepts

Speculative decoding Prefix cache Prefill and decode

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: phase decomposition + cache saves prefill + speculation saves decode + orthogonality argument + composition in RAG + monitoring each phase.

Real-world examples

Real products, models, and research that use this idea.

vLLM v1 ships prefix caching and speculative decoding as independent flags; the docs explicitly recommend enabling both for RAG workloads.
Anthropic's prompt caching (Claude Opus 4.7, Sonnet 4.6) sits on top of internal speculative decoding; the documented latency improvements are additive on long-prompt routes.
TensorRT-LLM Medusa and EAGLE-3 speculative decoding integrate with NVIDIA's KV-cache reuse layer and report stacking speedups on long-context inference.
DeepSeek V4 uses Multi-Token Prediction speculation and prefix caching together; the cost-per-token reductions reported come from combining both rather than choosing.
Together AI's serving stack lets users toggle prefix cache and speculation independently; their benchmarks show the two effects are roughly additive on decode-heavy traffic.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the formal speedup formula for speculative decoding, and which terms matter most?

QHow does prefix-cache invalidation interact with multi-tenant serving?

QCould speculation ever hurt a cache-hit request relative to no speculation?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming that a cache hit reduces the value of speculation because total request time is shorter. The relative speedup of speculation is set by decode-step cost and acceptance rate, neither of which the cache touches.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.