What does the first forward pass set up that the second one doesn't pay for?

Short answer·Medium·4.0 · 0·~3 min·Asked atBcg Pinecone Wandb·Relevant atNVIDIA

Attempt it

Production serving stacks run a warm up forward pass at the expected shapes before the pod accepts traffic. Explain what concretely happens during that first pass that the second and subsequent passes get for free, and why skipping warm up shows up as a multi-second TTFT spike on the very first real request.

Free · 2 AI evals / day

The first time a model runs on a GPU, several layers of the software stack do one-time setup work that the second and later passes do not pay for. A warm-up pass is just running a fake request through the model before traffic arrives, so that setup work happens on synthetic input instead of on a real user. Three things in particular fire on the first call. First, cuBLAS and cuDNN, the low-level math libraries, choose which algorithm to use for each matrix multiply. They support several algorithms per operation and pick based on the input shapes and the GPU. The first time they see a given shape they run a small search to pick the fastest one and cache that decision. The second time, they look it up. Second, CUDA graphs may be captured. A CUDA graph records the exact sequence of kernel launches the model performs and replays it on later calls without paying the per-launch CPU overhead. Building that graph requires actually running the model once. Third, any compiled stack (torch.compile, TensorRT-LLM, XLA) does just-in-time compilation of fused kernels the first time they are called. That compile can take seconds for a transformer-sized model. If you skip warm-up, all of that lands on the first real user request. Instead of the normal TTFT of a few hundred milliseconds, the first user might wait several seconds while the system finishes setting itself up. The standard practice is to run a synthetic batch at the expected shape before the pod accepts traffic, so the first real request hits a fully warmed-up runtime.

Production serving stacks do not run the first forward pass cleanly. The CUDA driver, cuBLAS, cuDNN, the framework runtime, and any compiler layered on top all defer per-shape work until they see real tensors. That work is one-shot and cached; it pays off on every subsequent pass. The cost of doing it lands wherever the first pass happens. The whole point of a warm-up phase is to make that first pass synthetic so users never see it. Specifically, four classes of work fire on the first pass at a given shape and dtype. First, cuBLAS and cuDNN run algorithm selection: each GEMM and convolution has several backend kernels (Tensor Core variants, split-K, different tile sizes), and the library benchmarks or heuristically picks the fastest one for the actual problem size. The result lives in a per-shape cache. Second, CUDA graph capture records the kernel launch sequence and lets the runtime replay it with one big driver call instead of one driver call per kernel. Capturing a graph requires executing the model once and freezing the launch pattern. Third, any JIT or AOT compiler in the stack builds its fused-kernel graph: torch.compile generates Triton kernels, TensorRT-LLM builds an engine, XLA emits a HLO graph, and these compiles can take from a few hundred milliseconds to many seconds depending on model size and op coverage. Fourth, the framework's caching memory allocator carves out its permanent pools and any tensor metadata caches get populated, so later passes do not allocate or page-fault. The cost is what surprises new operators. On a fresh pod, a transformer model can take 500 ms to 5 s of one-shot work on the first real forward pass, completely separate from the model weight load. That shows up as a single multi-second TTFT outlier the moment the load balancer points its first user at the pod.

TL;DR

The first forward pass runs cuBLAS/cuDNN heuristic search, CUDA-graph capture, and any JIT or TensorRT engine build, all cached after.

Memory aid

Easy to grasp

Imagine a chef arriving at a new kitchen for the first time. Before they can cook the first dish quickly, they have to learn where every pot lives, sharpen the knives to the right edge for this style of food, and pre-heat each burner to the temperature this menu wants. Once they have done all that, every dish after the first comes out fast. If you make a paying customer the first order of the night, that customer waits while the chef learns the kitchen. A warm-up is sending a fake order before opening so the chef can do all that learning, then the first real customer gets the fast experience.

Key concepts

Warm-up forward pass CUDA graphs Kernel autotune Time to first token Cold-start latency

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The first forward pass is special. Almost every layer of the GPU software stack defers shape-dependent work until real tensors arrive, then caches the result. The cost of that deferred work lands on whichever forward pass happens to be the first one. In production, that should be a synthetic warm-up batch, not a real user request.

This deep dive enumerates the caches that fill, quantifies the cost of each, explains why the caches are shape-keyed and what that means for warm-up coverage, and walks through the operational protocol that makes this invisible to users.

The four caches that fill on the first pass

Step through the layers of the stack and each one has its own one-shot cost.

cuBLAS and cuDNN heuristic selection. Both libraries expose multiple algorithm variants for each operation. cublasLt has dozens of GEMM algorithms differing in tile size, split-K strategy, and Tensor Core path. cudnnFind picks among convolution algorithms. The first time the library sees a problem size, it runs a brief benchmark or applies a learned heuristic to pick the fastest variant for that shape, dtype, and GPU. The choice goes into a cache keyed on the input parameters. Subsequent calls at the same shape skip the search and use the cached choice. In a transformer this cache fills across the QKV projection, the attention matmuls, the output projection, and the MLP gates; each distinct shape requires its own entry.

CUDA graph capture. A CUDA graph is a recorded sequence of kernel launches plus their parameters plus dependencies. After capture, the entire graph replays with a single cudaGraphLaunch call, eliminating the 5-20 microseconds of CPU overhead each unrecorded launch carries. For a 70B model with thousands of kernel launches per decode step, that overhead saving is several milliseconds per token. Capture requires executing the model once with stream-capture enabled. Because the graph captures specific pointers and shapes, it must be re-captured if those change, which is why continuous-batching stacks like vLLM pre-capture several graphs at the batch sizes they round forward passes to.

JIT or AOT compilation. Any compiler in the stack does serious work on the first call. torch.compile traces with TorchDynamo and lowers with Inductor to Triton-compiled fused kernels; the first call at each new shape takes seconds to compile, and a fresh compile on a 70B model can take a minute. TensorRT-LLM builds an engine, which is an exhaustive plan over fused operators, kernel choices, and tactic selection; the build itself happens ahead of time, but loading the engine and materializing plans on the device still happens at pod start. XLA does similar HLO compilation.

Allocator pools and metadata caches. PyTorch's caching allocator carves blocks lazily; the first allocation of each size hits the underlying cudaMalloc, which is slow and synchronizes the device. Once the pool is sized to the workload's working set, the allocator hands out reused blocks without driver calls. A warm-up pass at the production shape forces these pools to size themselves correctly, so steady-state requests never see an allocator stall.

Why caches are shape-keyed and what that means for coverage

Total cost: what skipping warm-up actually buys you

The operational protocol: probes, shapes, and timing

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Phase	What happens	Typical cost	Cache-key
Weight load	Read parameters from disk to HBM	1-30 s depending on size	Per pod
cuBLAS / cuDNN autotune	Benchmark or heuristic-pick best GEMM algorithm	10-200 ms per shape	Per (shape, dtype, GPU)
CUDA graph capture	Record kernel launch sequence	10-100 ms per graph	Per (shape, batch, seq_len)
JIT / TensorRT compile	Build fused-kernel engine	1-300 s on first ever build	Per (model, ops, shapes)
Allocator priming	Carve permanent memory pools	Tens of ms	Per pod
GPU clock ramp	DVFS reaches boost clocks	Sub-second	Per workload start

Real products, models, and research that use this idea.

vLLM ships a warmup_steps parameter that walks the configured decode batch sizes during engine init, before the API server starts accepting requests.
TensorRT-LLM builds an engine ahead of time, but its runtime still needs a warm-up forward pass on the GPU to materialize plans and prime allocators after loading the engine.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy must warm-up cover all production shapes, not just one?

cuBLAS, cuDNN, CUDA graphs, and TensorRT all key their caches on shape (and often on dtype and stride). A model warmed at batch 1 and prompt length 128 will hit a fresh autotune the first time batch 8 or prompt length 4096 arrives. Production warm-up routines enumerate the shape grid: typical batches and a coarse prompt-length spectrum.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating cold start as a model-load problem. The weight load is part of it, but the bigger surprise is the per-shape autotune, graph capture, and JIT compile that fire only on the first forward pass at each shape.

60 second bullets to scan on the way to the call.

What cuBLAS and cuDNN cache on first call and why it is shape-keyed
What CUDA graphs are and why capture is slow but replay is fast

Primary sources. Browse if you want the original framing.