Walk through how CUDA graphs reshape per step decode latency and when the win disappears

Every time the GPU is asked to run a kernel, the CPU does a little setup work: it builds a launch packet (which kernel, which arguments, which streams), tells the driver to enqueue it, and the driver eventually hands it to the GPU. That CPU-side setup costs roughly 10-30 microseconds per kernel. One decode step fires many kernels: layer norm, query/key/value projections, attention, output projection, MLP up-projection, activation, down-projection, residual adds, and so on. Across all layers that is hundreds of kernel launches per token. If each kernel runs in 10-50 microseconds (which is what happens at small batch, when decode is bandwidth-bound), the total launch overhead per step can be comparable to or larger than the actual GPU work. The GPU sits idle waiting for the next launch. CUDA graphs solve this. You execute one decode step in a special 'capture mode' that records all the launches. After that, instead of launching kernels one by one, you submit the entire recorded graph as a single operation. The GPU runs the same kernels but the CPU dispatch happens essentially once. At small batch this can give a 1.2-1.5x speedup on decode. The catch: CUDA graphs assume the launch sequence stays the same. If the batch shape changes (different number of requests, different sequence lengths), the recorded graph is no longer valid and you have to recapture. Modern serving stacks handle this by padding to a small set of fixed shapes (shape bucketing) and capturing one graph per bucket.

Kernel launch overhead is real and concentrated on the CPU side. Each launch costs about 10-30 microseconds for the driver to build a launch packet, push it through the queue, and schedule it on the GPU. A transformer decode step launches dozens to hundreds of kernels (attention, projections, MLP, norms, sampling, residuals), so the cumulative launch cost per step can run into milliseconds. At small batch, decode is bandwidth-bound and individual kernels are short. A GEMM that streams an 80 GB weight stack at 3 TB/s of HBM bandwidth runs in about 25 ms total for the whole step, but it is split across many small matmuls and attention kernels each in the 20-100 us range. When the kernel itself runs in tens of microseconds and the launch costs tens of microseconds, the GPU is idle for a meaningful fraction of every step. Profilers show characteristic gaps between adjacent kernels. CUDA graphs record the entire decode step (all kernel launches with their arguments and stream dependencies) into a single graph object during a one-time capture pass. Subsequent decode steps replay the captured graph with one submission, so the per-launch CPU cost is paid essentially once per step rather than once per kernel. The kernels themselves run at the same speed; what disappears is the CPU-to-GPU dispatch overhead between them. The speedup is workload-dependent and concentrated at small batch. At batch size 1 on a small model, CUDA graphs can deliver 1.5-2x decode throughput improvements. At batch size 64 on a 70B model, each kernel runs long enough that launch overhead is a few percent of the step, and CUDA graphs add only marginal improvement. Two limitations matter in production. First, CUDA graphs require fixed launch shapes: any variation in tensor shapes, kernel arguments, or branching forces a recapture. Continuous batching introduces this problem directly because every step composes a different mix of requests.