Prefix cache hit lands on a request: does speculative decoding still help that request?

**The two techniques target different phases of the request.** A request to an LLM has two phases: prefill (process the entire prompt in one parallel pass) and decode (generate the response one token at a time). They have different bottlenecks and respond to different optimizations. **Prefix cache** saves prefill work. When the serving system has already processed the same prompt prefix for an earlier request, it has already computed and stored the KV-cache entries for that span. A cache hit means the prefill step reuses those entries and only computes the tail beyond the cached region. This can drop prefill from hundreds of milliseconds to a few milliseconds on long-context workloads like RAG over a shared knowledge base. **Speculative decoding** saves decode work. A small draft model (or extra heads on the target model) proposes several candidate next tokens. The target model then verifies those candidates in a single forward pass and accepts the ones that match its own distribution. On a successful acceptance run, you got multiple tokens for one HBM sweep of the target's weights, which is the dominant cost in memory-bandwidth-bound decode. So the cache touches phase A and speculation touches phase B. A cache hit changes how long prefill takes; it does not change how long each decode step takes or what acceptance rate the draft model achieves. Speculation gives the same relative win whether prefill was cached or not. In practice the two compose well. Long-prefix decode-heavy workloads (RAG, long system prompts plus medium completions) get both. The cache cuts the prefill tax; speculation cuts the decode tax; total request latency drops on both fronts.

**They attack different halves of the request, so they stack.** Prefix cache reduces prefill by reusing KV entries for a shared prompt prefix. Decode is unchanged: the model still generates one token at a time, still reads the entire KV cache from HBM each step, and is still memory-bandwidth bound exactly as before. The cache hit shrinks the front of the request, not the per-token step. Speculative decoding lives entirely inside the decode loop. A draft model proposes several candidate tokens; the target verifies them in one forward pass; accepted candidates yield multiple tokens per HBM sweep of the target's weights. The speedup is a function of decode-step cost times acceptance rate, both intrinsic to the decode loop and the target/draft pair. Neither quantity changes when prefill happens to be cached. **Conclusion: speculation's relative win on a cache-hit request equals its relative win on a cache-miss request.** The absolute latency improvement is in the decode segment, which is the same segment in both cases. Total request time differs only in the prefill prefix, which the cache hit shortened. The two compose cleanly in long-prefix decode-heavy workloads. RAG over a shared knowledge base has long, repeating prefixes (the cache wins) and meaningful response lengths (speculation wins). A code-assistant with long system prompts is the same shape. The right operational posture: enable both, tune acceptance rate independently of cache hit rate, monitor each savings metric separately on its own phase.