What does the first forward pass set up that the second one doesn't pay for?

The first time a model runs on a GPU, several layers of the software stack do one-time setup work that the second and later passes do not pay for. A warm-up pass is just running a fake request through the model before traffic arrives, so that setup work happens on synthetic input instead of on a real user. Three things in particular fire on the first call. First, cuBLAS and cuDNN, the low-level math libraries, choose which algorithm to use for each matrix multiply. They support several algorithms per operation and pick based on the input shapes and the GPU. The first time they see a given shape they run a small search to pick the fastest one and cache that decision. The second time, they look it up. Second, CUDA graphs may be captured. A CUDA graph records the exact sequence of kernel launches the model performs and replays it on later calls without paying the per-launch CPU overhead. Building that graph requires actually running the model once. Third, any compiled stack (torch.compile, TensorRT-LLM, XLA) does just-in-time compilation of fused kernels the first time they are called. That compile can take seconds for a transformer-sized model. If you skip warm-up, all of that lands on the first real user request. Instead of the normal TTFT of a few hundred milliseconds, the first user might wait several seconds while the system finishes setting itself up. The standard practice is to run a synthetic batch at the expected shape before the pod accepts traffic, so the first real request hits a fully warmed-up runtime.

Production serving stacks do not run the first forward pass cleanly. The CUDA driver, cuBLAS, cuDNN, the framework runtime, and any compiler layered on top all defer per-shape work until they see real tensors. That work is one-shot and cached; it pays off on every subsequent pass. The cost of doing it lands wherever the first pass happens. The whole point of a warm-up phase is to make that first pass synthetic so users never see it. Specifically, four classes of work fire on the first pass at a given shape and dtype. First, cuBLAS and cuDNN run algorithm selection: each GEMM and convolution has several backend kernels (Tensor Core variants, split-K, different tile sizes), and the library benchmarks or heuristically picks the fastest one for the actual problem size. The result lives in a per-shape cache. Second, CUDA graph capture records the kernel launch sequence and lets the runtime replay it with one big driver call instead of one driver call per kernel. Capturing a graph requires executing the model once and freezing the launch pattern. Third, any JIT or AOT compiler in the stack builds its fused-kernel graph: torch.compile generates Triton kernels, TensorRT-LLM builds an engine, XLA emits a HLO graph, and these compiles can take from a few hundred milliseconds to many seconds depending on model size and op coverage. Fourth, the framework's caching memory allocator carves out its permanent pools and any tensor metadata caches get populated, so later passes do not allocate or page-fault. The cost is what surprises new operators. On a fresh pod, a transformer model can take 500 ms to 5 s of one-shot work on the first real forward pass, completely separate from the model weight load. That shows up as a single multi-second TTFT outlier the moment the load balancer points its first user at the pod.