Explain what CUDA graphs do during decode and why the speedup is most visible at small batch. Then identify two conditions under which CUDA graphs help less, or break entirely.
Free · 2 AI evals / day
Every time the GPU is asked to run a kernel, the CPU does a little setup work: it builds a launch packet (which kernel, which arguments, which streams), tells the driver to enqueue it, and the driver eventually hands it to the GPU. That CPU-side setup costs roughly 10-30 microseconds per kernel.
One decode step fires many kernels: layer norm, query/key/value projections, attention, output projection, MLP up-projection, activation, down-projection, residual adds, and so on. Across all layers that is hundreds of kernel launches per token.
If each kernel runs in 10-50 microseconds (which is what happens at small batch, when decode is bandwidth-bound), the total launch overhead per step can be comparable to or larger than the actual GPU work. The GPU sits idle waiting for the next launch.
CUDA graphs solve this. You execute one decode step in a special 'capture mode' that records all the launches. After that, instead of launching kernels one by one, you submit the entire recorded graph as a single operation. The GPU runs the same kernels but the CPU dispatch happens essentially once. At small batch this can give a 1.2-1.5x speedup on decode.
The catch: CUDA graphs assume the launch sequence stays the same. If the batch shape changes (different number of requests, different sequence lengths), the recorded graph is no longer valid and you have to recapture. Modern serving stacks handle this by padding to a small set of fixed shapes (shape bucketing) and capturing one graph per bucket.
Kernel launch overhead is real and concentrated on the CPU side. Each launch costs about 10-30 microseconds for the driver to build a launch packet, push it through the queue, and schedule it on the GPU. A transformer decode step launches dozens to hundreds of kernels (attention, projections, MLP, norms, sampling, residuals), so the cumulative launch cost per step can run into milliseconds.
At small batch, decode is bandwidth-bound and individual kernels are short. A GEMM that streams an 80 GB weight stack at 3 TB/s of HBM bandwidth runs in about 25 ms total for the whole step, but it is split across many small matmuls and attention kernels each in the 20-100 us range. When the kernel itself runs in tens of microseconds and the launch costs tens of microseconds, the GPU is idle for a meaningful fraction of every step. Profilers show characteristic gaps between adjacent kernels.
CUDA graphs record the entire decode step (all kernel launches with their arguments and stream dependencies) into a single graph object during a one-time capture pass. Subsequent decode steps replay the captured graph with one submission, so the per-launch CPU cost is paid essentially once per step rather than once per kernel. The kernels themselves run at the same speed; what disappears is the CPU-to-GPU dispatch overhead between them.
The speedup is workload-dependent and concentrated at small batch. At batch size 1 on a small model, CUDA graphs can deliver 1.5-2x decode throughput improvements. At batch size 64 on a 70B model, each kernel runs long enough that launch overhead is a few percent of the step, and CUDA graphs add only marginal improvement.
Two limitations matter in production. First, CUDA graphs require fixed launch shapes: any variation in tensor shapes, kernel arguments, or branching forces a recapture. Continuous batching introduces this problem directly because every step composes a different mix of requests.
Quick reference
TL;DR
CUDA graphs record the kernel-launch sequence of one decode step and replay the whole graph as a single submission, collapsing dozens of per-step CPU dispatches into one.
Picture a busy chef shouting instructions to a fast cook for each tiny task: chop, stir, pour, taste. The cook is so fast that most of the wall-clock time is the chef finishing one sentence and starting the next. CUDA graphs let the chef write the whole recipe down once and just say 'do that recipe' on every batch, so all the talking happens once. This makes a big difference when the cook is much faster than the chef can talk, which is exactly what happens during decode at small batch. But if the recipe changes every batch (different ingredients, different steps), writing it down does not help, and that is when CUDA graphs stop working.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Large-batch serving where each decode step is already long (50+ ms) and launch overhead is a small fraction The marginal speedup is single-digit percent and the GPU memory cost of holding captured graphs is not worth it.
Workloads with highly variable per-step shapes that resist bucketing (e.g. extreme long-tail sequence lengths) You either pay recapture cost continuously or maintain so many graphs that GPU memory is exhausted.
Prefill phase, which processes whole prompts in one parallel pass with high arithmetic intensity Prefill kernels are large and compute-bound; launch overhead is already a tiny fraction of the step, so CUDA graphs add little.
MoE serving with per-token expert routing changing the kernel sequence Data-dependent kernel choice breaks graph capture; you would need per-route graphs which explodes combinatorially.
Quick scan · 2–4 min
Everything important, quickly.
3 min: explain the ~10-30 us per-launch CPU overhead, count kernels per decode step, identify the small-batch regime where overhead dominates, describe what a CUDA graph captures and replays, then name the large-batch erosion and the shape-variability break with shape bucketing as the standard mitigation.
Side-by-side
Regime
Per-step time without graphs
Per-step time with graphs
Win
Small batch (1-4) on 7B-13B model
~3-8 ms (mostly dispatch)
~1.5-4 ms
1.5-2x
Small batch on 70B model
~25-30 ms (dispatch is ~10-20% of step)
~22-26 ms
1.1-1.3x
Large batch (32-64) on 70B
~50-60 ms (dispatch <5%)
~48-58 ms
<1.05x
Variable-shape continuous batching
Per-step recapture cost
Per-bucket graphs + padding
Depends on bucket coverage
Real-world examples
Real products, models, and research that use this idea.
1vLLM captures CUDA graphs per (batch_size, max_seq_len) bucket and routes each scheduling step to the nearest bucket via padding.
2TensorRT-LLM's runtime ships explicit CUDA-graph capture for decode with shape buckets defined at engine build time.
3SGLang uses CUDA graphs for the decode loop and falls back to eager execution for prefill, where shapes are highly variable.
4NVIDIA's H100 and B200 driver releases include latency improvements specifically targeting per-launch overhead, but CUDA graphs remain the standard way to eliminate it for repeated decode steps.
5PyTorch 2.x exposes cudagraph trees as a higher-level abstraction that handles capture and replay automatically for static graphs.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does CUDA graph replay differ from compiler-level kernel fusion?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the captured graph need fixed tensor shapes?
Sign in or go Pro to see the approach for this follow-up.
QHow does shape bucketing trade GPU memory for serving flexibility?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking CUDA graphs make the GPU work faster. The kernels themselves run at the same speed; CUDA graphs only remove the CPU-side dispatch overhead between them. The win comes from eliminating idle gaps, not from accelerating compute.