Speculative decoding is already on and batch size just climbed to 64, keep it or kill it?

MCQ·Medium·4.0 · 0·~1 min·Asked atDescript Mongodb Snap·Relevant atNVIDIA

Attempt it

**The answer is A: kill it.** Speculative decoding trades extra GPU work for lower wall-clock latency. The trade is profitable when the GPU has spare work to do, and unprofitable when it does not. **The mechanics.** A small draft model proposes K tokens at a time. The full target model verifies all K tokens in a single forward pass. If most drafts match what the target would have produced, the system commits multiple tokens per target forward pass. Wall time per stream drops. **Why batch matters.** At batch 1, the target model is bandwidth-bound on a single stream and has lots of spare compute. Verifying K extra tokens is almost free because the same weights have to be read regardless. Latency drops by 1.5-2.5x typically. **At batch 64**, the target is already running near peak bandwidth across all 64 streams. Verifying K extra draft tokens per stream means reading K+1 sets of activations through the same bandwidth pipe. The verify pass now competes with the steady decode work. Throughput drops below baseline (no spec) and latency per stream gets worse, not better. **The production pattern.** Serving stacks measure a batch-size threshold (often around batch 16-32 depending on hardware) and disable spec decoding above it. Below the threshold, spec is on. Above, off.

**Kill it.** At batch 64, speculative decoding has become a net loss. The framework that justified spec at small batch (spare bandwidth headroom to verify extra tokens for free) no longer holds. **The two regimes.** At **small batch** (1-4), decode is latency-bound on per-stream bandwidth. The target model reads its full weight matrix once per step regardless of whether one or several tokens are being verified. K draft tokens proposed and verified in a single forward pass deliver K+1 candidate tokens per step. With 70-80% acceptance, that is 2-3x wall-clock speedup per stream. At **large batch** (32, 64+), decode is throughput-bound. The target is already saturating HBM bandwidth serving all concurrent streams. Extra draft tokens per stream means the verify pass moves more data per stream, but bandwidth is fully consumed. Net effect: verify steals cycles from real decode work, aggregate throughput drops. **Why option B is wrong.** Acceptance rate is roughly batch-independent (the draft model's prediction quality does not change). But per-token speedup is the wrong metric in the throughput regime. The right metric is aggregate tokens per second across the cluster, and that metric falls at batch 64 with spec on. **Why option C is wrong.** Raising draft length makes the verify pass more expensive, deepening the bandwidth contention at large batch. It works in the opposite direction of what is needed. **Why option D is wrong.** A CPU side-channel draft does not solve the problem because the contention is on the target GPU's HBM, not on draft compute. Draft compute is already cheap; the issue is verify cost on the target side. **The production pattern.** Serving stacks like vLLM, TensorRT-LLM, and SGLang implement a batch-size threshold for spec decoding. Below the threshold (typically 16-32), spec is enabled. Above, it is disabled. The threshold is measured empirically on the deployment hardware against representative workloads.

TL;DR

Kill it. At batch 64, decode is already bandwidth-bound and the verify pass competes for the same HBM cycles. Spec decoding helps only when there is spare bandwidth headroom.

Memory aid

Easy to grasp

Picture a single delivery truck that can carry 64 boxes. Normally each box is one customer's order. Someone proposes a clever trick: send the driver to grab three extra speculative boxes along the route, in case the customer wants more. At an empty truck (1 order), this is great because there is plenty of room and the trip is short. At a full truck (64 orders), the extras crowd out real boxes and the trip takes longer. The trick only worked when there was slack capacity. Speculative decoding behaves the same way. Small batch has slack bandwidth, so verifying extra draft tokens is free. Large batch has no slack, so verifying the drafts steals room from real requests.

Key concepts

Speculative decoding Batch size Memory bandwidth

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Speculative decoding is one of the most powerful per-stream latency optimizations available in LLM serving, but it is conditional on having spare GPU bandwidth to spend. At low batch, the spare bandwidth is abundant and spec decoding cuts wall time by 1.5-3x per stream. At large batch, the bandwidth is fully consumed by the steady decode work, and spec decoding becomes a net loss.

The transition is sharp. A serving stack tuned for low-traffic latency benefits flips to a throughput sink the moment the workload pushes batch above the crossover threshold. Production deployments that fail to gate spec decoding on batch size pay a quiet throughput penalty at scale, often without realizing it.

This deep dive covers the mechanics of speculative decoding, why the bandwidth headroom argument works at low batch, why it inverts at large batch, the role of acceptance rate and draft length in the calculation, and the production pattern of batch-size gating. By the end you should be able to recognize when spec decoding helps, when it hurts, and how to operationalize the decision.

How speculative decoding actually works

Speculative decoding pairs a small draft model (e.g., 1B parameters) with a large target model (e.g., 70B parameters). The draft proposes K future tokens (K typically 4-8) by running a quick autoregressive sequence. The target then runs a single forward pass that verifies all K positions in parallel.

Verification works because attention is causal: the target can compute its own predicted tokens at positions t+1, t+2, ..., t+K conditioned on a hypothetical sequence that includes the draft's proposed tokens. If the target's prediction at position t+i matches the draft's proposal at position t+i, that token is accepted. The first mismatch terminates acceptance; the system commits the accepted prefix and the target's own prediction at the mismatch point.

In the best case (all K drafts accepted), the system advances K+1 tokens per target forward pass. In the worst case (first draft rejected), it advances 1 token (the target's own at position t+1). Average advance with 70-80% acceptance lands around 3-4 tokens per step for K=4.

The target forward pass is the dominant cost. Draft compute is typically 1-3% of target compute, so the draft model is essentially free relative to the verify.

Why spec wins at small batch

Why spec loses at large batch

Acceptance rate, draft length, and threshold tuning

Production patterns and serving-stack support

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM exposes a speculative_max_batch parameter to disable spec above a tuned threshold.
TensorRT-LLM speculative decoding includes batch-size gating in its release notes for Hopper deployments.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does draft model size affect the batch-size threshold?

A larger draft model has higher acceptance rate (better prediction) but more draft compute cost. The net effect can extend or contract the profitable batch range. Very small drafts (e.g., 1B for a 70B target) shift the threshold higher because draft cost is negligible; larger drafts (7B for 70B target) shift it lower because draft cost itself starts to compete.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reasoning about acceptance rate at large batch. The acceptance rate may be identical, but the verify pass now steals bandwidth from concurrent decodes, erasing the per-stream win in aggregate.

60 second bullets to scan on the way to the call.

Describe the bandwidth-headroom argument for spec decoding at small batch
Explain why the same argument fails at large batch