How CUDA graphs reshape per step decode latency and when the win disappears
Explain what CUDA graphs do during decode and why the speedup is most visible at small batch. Then identify two conditions under which CUDA graphs help less, or break entirely.
CUDA graphs record the kernel-launch sequence of one decode step and replay the whole graph as a single submission, collapsing dozens of per-step CPU dispatches into one.
Picture a busy chef shouting instructions to a fast cook for each tiny task: chop, stir, pour, taste. The cook is so fast that most of the wall-clock time is the chef finishing one sentence and starting the next. CUDA graphs let the chef write the whole recipe down once and just say 'do that recipe' on every batch, so all the talking happens once. This makes a big difference when the cook is much faster than the chef can talk, which is exactly what happens during decode at small batch. But if the recipe changes every batch (different ingredients, different steps), writing it down does not help, and that is when CUDA graphs stop working.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
CUDA graphs are a CPU-side optimization with a large effect on LLM decode latency. They do not make the GPU faster at anything. They remove the wall-clock time the GPU spends waiting for the next kernel to be dispatched.
GPU work and GPU dispatch are separate costs. The GPU itself can execute kernels back to back with very little overhead once they are queued, but each kernel needs to be queued by the host CPU, and that dispatch has a non-trivial cost (10-30 microseconds on typical hardware). For workloads with many short kernels, the dispatch can become a larger share of wall-clock time than the actual compute.
LLM decode at small batch is exactly that workload. Each step fires hundreds of kernels, each kernel runs in tens of microseconds, and the GPU spends measurable time idle waiting for the next launch. CUDA graphs record the whole launch sequence once and replay it as a single submission, which turns the total dispatch overhead into a near-constant cost per step. In production this brings its own costs: shape bucketing, graph memory, and cases where the gain is small.
Where the overhead comes from
When the host CPU calls a kernel launch (e.g. via a CUDA runtime API or a PyTorch op), several things have to happen. The runtime builds a launch packet: which kernel binary to invoke, which arguments (including pointers, scalars, shapes), which grid and block dimensions, which stream. It checks for any pending dependencies. It enqueues the packet through the driver, which may involve a system call or shared-memory write to the kernel-launch ring buffer. Eventually the GPU's host interface reads the packet, validates it, and schedules the kernel on a streaming multiprocessor.
Most of this is fast in isolation, but the cumulative latency from the moment user code calls cudaLaunchKernel to the moment the GPU starts executing is roughly 10-30 microseconds on a typical system, counting framework and driver work. It is higher under host CPU pressure or with profiler hooks, and lower with a fast, idle CPU. A bare CUDA launch from C++ costs only a few microseconds; Python and framework layers add most of the rest.
Launches are asynchronous. The CPU can queue the next kernels while the GPU runs the current one, so dispatch costs time only when kernels finish faster than the CPU can issue new ones.
For compute-bound workloads with long kernels (e.g. prefill on a large model, training step at large batch), this cost is hidden. The kernel itself runs for milliseconds, and a 20-microsecond dispatch is negligible.
For bandwidth-bound workloads with short kernels (decode at small batch), this can be the largest cost. A decode step on a 70B model launches hundreds of kernels across layer norms, qkv projections, attention, output projection, MLP layers, residuals, and sampling. At 10-30 us each, the total dispatch can be several milliseconds per token, comparable to the GPU's own time at small batch.
In a profiler this shows up as gaps between adjacent kernels on the GPU timeline: the GPU finishes one kernel, sits idle for tens of microseconds, then starts the next. A continuous, gap-free GPU timeline indicates you are either compute-bound (kernels are long enough to overlap dispatch) or already using CUDA graphs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Regime | Per-step time without graphs | Per-step time with graphs | Win |
|---|---|---|---|
| Batch 1, 13B model in BF16 on one H100 | About 13-14 ms (8-9 ms weight read plus 5 ms of launch gaps) | About 9 ms | About 1.4-1.5x |
| Batch 32 at 1,024-token contexts, same model | Up to about 22.6 ms (17.6 ms weight and KV reads plus up to 5 ms of gaps) | About 17.6 ms | Up to about 1.3x |
| Large batch with long contexts, or compute-bound steps | Long step; launch gaps a small share | Slightly shorter step | Small |
| Variable-shape continuous batching | Per-step recapture cost | Per-bucket graphs + padding | Depends on bucket coverage |
Real products, models, and research that use this idea.
- vLLM captures one CUDA graph per batch size in a list (1, 2, 4, then steps of 8 and 16, capped at 512 by default) and pads each batch up to the nearest captured size; its default mode uses full graphs for decode and piecewise graphs for other batches.
- TensorRT-LLM's PyTorch backend captures decode CUDA graphs for a list of batch sizes set in CudaGraphConfig, and can pad batches up to the nearest captured size.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does CUDA graph replay differ from compiler-level kernel fusion?
Fusion reduces the number of kernels by merging adjacent ones into one larger kernel, so total launches drop. CUDA graphs keep the same kernels but remove the per-launch CPU overhead between them. The two work together: fuse what you can to reduce kernel count, then graph-capture what remains to remove the dispatch between them.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking CUDA graphs make the GPU work faster. The kernels themselves run at the same speed; CUDA graphs only remove the CPU-side dispatch overhead between them. The win comes from removing idle gaps, not from faster compute.
60 second bullets to scan on the way to the call.
Why kernel launch has a CPU-side cost (~10-30 us per launch)
Why decode at small batch is dominated by launch overhead
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.