Speculative decoding is already on and batch size just climbed to 64, keep it or kill it?

**The answer is A: kill it.** Speculative decoding trades extra GPU work for lower wall-clock latency. The trade is profitable when the GPU has spare work to do, and unprofitable when it does not. **The mechanics.** A small draft model proposes K tokens at a time. The full target model verifies all K tokens in a single forward pass. If most drafts match what the target would have produced, the system commits multiple tokens per target forward pass. Wall time per stream drops. **Why batch matters.** At batch 1, the target model is bandwidth-bound on a single stream and has lots of spare compute. Verifying K extra tokens is almost free because the same weights have to be read regardless. Latency drops by 1.5-2.5x typically. **At batch 64**, the target is already running near peak bandwidth across all 64 streams. Verifying K extra draft tokens per stream means reading K+1 sets of activations through the same bandwidth pipe. The verify pass now competes with the steady decode work. Throughput drops below baseline (no spec) and latency per stream gets worse, not better. **The production pattern.** Serving stacks measure a batch-size threshold (often around batch 16-32 depending on hardware) and disable spec decoding above it. Below the threshold, spec is on. Above, off.

**Kill it.** At batch 64, speculative decoding has become a net loss. The framework that justified spec at small batch (spare bandwidth headroom to verify extra tokens for free) no longer holds. **The two regimes.** At **small batch** (1-4), decode is latency-bound on per-stream bandwidth. The target model reads its full weight matrix once per step regardless of whether one or several tokens are being verified. K draft tokens proposed and verified in a single forward pass deliver K+1 candidate tokens per step. With 70-80% acceptance, that is 2-3x wall-clock speedup per stream. At **large batch** (32, 64+), decode is throughput-bound. The target is already saturating HBM bandwidth serving all concurrent streams. Extra draft tokens per stream means the verify pass moves more data per stream, but bandwidth is fully consumed. Net effect: verify steals cycles from real decode work, aggregate throughput drops. **Why option B is wrong.** Acceptance rate is roughly batch-independent (the draft model's prediction quality does not change). But per-token speedup is the wrong metric in the throughput regime. The right metric is aggregate tokens per second across the cluster, and that metric falls at batch 64 with spec on. **Why option C is wrong.** Raising draft length makes the verify pass more expensive, deepening the bandwidth contention at large batch. It works in the opposite direction of what is needed. **Why option D is wrong.** A CPU side-channel draft does not solve the problem because the contention is on the target GPU's HBM, not on draft compute. Draft compute is already cheap; the issue is verify cost on the target side. **The production pattern.** Serving stacks like vLLM, TensorRT-LLM, and SGLang implement a batch-size threshold for spec decoding. Below the threshold (typically 16-32), spec is enabled. Above, it is disabled. The threshold is measured empirically on the deployment hardware against representative workloads.