Same topic, related formats. Practice these next.
Production serving stacks run a warm-up forward pass at the expected shapes before the pod accepts traffic. Explain what concretely happens during that first pass that the second and subsequent passes get for free, and why skipping warm-up shows up as a multi-second TTFT spike on the very first real request.
The first forward pass runs cuBLAS/cuDNN heuristic search, CUDA-graph capture, and any JIT or TensorRT engine build, all cached after.
Imagine a chef arriving at a new kitchen for the first time. Before they can cook the first dish quickly, they have to learn where every pot lives, sharpen the knives to the right edge for this style of food, and pre-heat each burner to the temperature this menu wants. Once they have done all that, every dish after the first comes out fast. If you make a paying customer the first order of the night, that customer waits while the chef learns the kitchen. A warm-up is sending a fake order before opening so the chef can do all that learning, then the first real customer gets the fast experience.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: name the four caches that fill on first pass (cuBLAS heuristic, CUDA graphs, JIT engine, allocator pools), quantify the total cost, explain shape-keyed caching, and connect to the readiness-probe placement.
| Phase | What happens | Typical cost | Cache-key |
|---|---|---|---|
| Weight load | Read parameters from disk to HBM | 1-30 s depending on size | Per pod |
| cuBLAS / cuDNN autotune | Benchmark or heuristic-pick best GEMM algorithm | 10-200 ms per shape | Per (shape, dtype, GPU) |
| CUDA graph capture | Record kernel launch sequence | 10-100 ms per graph | Per (shape, batch, seq_len) |
| JIT / TensorRT compile | Build fused-kernel engine | 1-300 s on first ever build | Per (model, ops, shapes) |
| Allocator priming | Carve permanent memory pools | Tens of ms | Per pod |
| GPU clock ramp | DVFS reaches boost clocks | Sub-second | Per workload start |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating cold start as a model-load problem. The weight load is part of it, but the bigger surprise is the per-shape autotune, graph capture, and JIT compile that fire only on the first forward pass at each shape.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.