**The answer is A: kill it.** Speculative decoding trades extra GPU work for lower wall-clock latency. The trade is profitable when the GPU has spare work to do, and unprofitable when it does not.
**The mechanics.** A small draft model proposes K tokens at a time. The full target model verifies all K tokens in a single forward pass. If most drafts match what the target would have produced, the system commits multiple tokens per target forward pass. Wall time per stream drops.
**Why batch matters.** At batch 1, the target model is bandwidth-bound on a single stream and has lots of spare compute. Verifying K extra tokens is almost free because the same weights have to be read regardless. Latency drops by 1.5-2.5x typically.
**At batch 64**, the target is already running near peak bandwidth across all 64 streams. Verifying K extra draft tokens per stream means reading K+1 sets of activations through the same bandwidth pipe. The verify pass now competes with the steady decode work. Throughput drops below baseline (no spec) and latency per stream gets worse, not better.
**The production pattern.** Serving stacks measure a batch-size threshold (often around batch 16-32 depending on hardware) and disable spec decoding above it. Below the threshold, spec is on. Above, off.
**Kill it.** At batch 64, speculative decoding has become a net loss. The framework that justified spec at small batch (spare bandwidth headroom to verify extra tokens for free) no longer holds.
**The two regimes.**
At **small batch** (1-4), decode is latency-bound on per-stream bandwidth. The target model reads its full weight matrix once per step regardless of whether one or several tokens are being verified. K draft tokens proposed and verified in a single forward pass deliver K+1 candidate tokens per step. With 70-80% acceptance, that is 2-3x wall-clock speedup per stream.
At **large batch** (32, 64+), decode is throughput-bound. The target is already saturating HBM bandwidth serving all concurrent streams. Extra draft tokens per stream means the verify pass moves more data per stream, but bandwidth is fully consumed. Net effect: verify steals cycles from real decode work, aggregate throughput drops.
**Why option B is wrong.** Acceptance rate is roughly batch-independent (the draft model's prediction quality does not change). But per-token speedup is the wrong metric in the throughput regime. The right metric is aggregate tokens per second across the cluster, and that metric falls at batch 64 with spec on.
**Why option C is wrong.** Raising draft length makes the verify pass more expensive, deepening the bandwidth contention at large batch. It works in the opposite direction of what is needed.
**Why option D is wrong.** A CPU side-channel draft does not solve the problem because the contention is on the target GPU's HBM, not on draft compute. Draft compute is already cheap; the issue is verify cost on the target side.
**The production pattern.** Serving stacks like vLLM, TensorRT-LLM, and SGLang implement a batch-size threshold for spec decoding. Below the threshold (typically 16-32), spec is enabled. Above, it is disabled. The threshold is measured empirically on the deployment hardware against representative workloads.
Quick reference
TL;DR
Kill it. At batch 64, decode is already bandwidth-bound and the verify pass competes for the same HBM cycles. Spec decoding helps only when there is spare bandwidth headroom.
Picture a single delivery truck that can carry 64 boxes. Normally each box is one customer's order. Someone proposes a clever trick: send the driver to grab three extra speculative boxes along the route, in case the customer wants more. At an empty truck (1 order), this is great because there is plenty of room and the trip is short. At a full truck (64 orders), the extras crowd out real boxes and the trip takes longer. The trick only worked when there was slack capacity. Speculative decoding behaves the same way. Small batch has slack bandwidth, so verifying extra draft tokens is free. Large batch has no slack, so verifying the drafts steals room from real requests.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Latency-sensitive single-stream serving Small batch decode has bandwidth headroom; spec decoding cuts per-stream wall time substantially and should be on.
Workloads with very high draft acceptance rates (>90%) High acceptance changes the cost calculus and can extend the profitable batch range; measure on the deployment hardware before disabling.
Workloads with cheap or asynchronous draft models If draft compute is fully overlapped with target verify, the cost equation shifts; some configurations remain profitable at moderately higher batches.
Pre-deployment performance tuning Always measure the actual cross-over batch on the production hardware; the rule of thumb threshold (16-32) varies with model size, draft model size, and HBM bandwidth.
Quick scan · 2–4 min
Everything important, quickly.
5 min: anchor spec decoding as a spare-bandwidth trade, contrast batch 1 (headroom-rich) versus batch 64 (saturated), walk through the verify cost growing linearly in K, and close with the production pattern of batch-size gating.
Real-world examples
Real products, models, and research that use this idea.
1vLLM exposes a speculative_max_batch parameter to disable spec above a tuned threshold.
2TensorRT-LLM speculative decoding includes batch-size gating in its release notes for Hopper deployments.
3DeepSeek V4 inference uses spec decoding at low batch but disables it for high-throughput batched serving.
4Anthropic Claude Opus 4.7 has been observed to use spec decoding selectively for low-latency single-stream paths.
5Together AI's Llama 4 Maverick endpoint runs spec decoding on its latency-tier offering but not on its throughput-tier offering.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does draft model size affect the batch-size threshold?
Sign in or go Pro to see the approach for this follow-up.
QWhat is medusa decoding and how does it change the speculative decoding tradeoff?
Sign in or go Pro to see the approach for this follow-up.
QWhy does spec decoding sometimes help at moderate batch even when the simple bandwidth argument says it should not?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reasoning about acceptance rate at large batch. The acceptance rate may be identical, but the verify pass now steals bandwidth from concurrent decodes, erasing the per-stream win in aggregate.