Which components dominate cold-start latency when a serving replica spins up?

Multi-select·Medium·4.0 · 0·~1 min·Relevant atDatabricks Microsoft Modal Labs

Attempt it

**Cold start** is the wall-clock time between an autoscaler deciding to add a replica and that replica being ready to serve its first request. For LLM serving it is usually measured in minutes, not seconds, and it walks through four phases that all happen every time. The first phase is pod scheduling and container image pull. Kubernetes (or whatever orchestrator) picks a node, the node pulls the serving runtime container from the registry, and the kubelet starts the container. This takes tens of seconds depending on image size and registry bandwidth. The second phase is the model-weight download. Weights live in object storage like S3 or GCS, and the replica streams them to its local disk on startup. For a 70B-class model this is tens of gigabytes and can take minutes. This phase is the usual bottleneck. The third phase loads the on-disk weights into GPU HBM. Once the bytes are on the local NVMe, copying them into device memory is a few seconds. The fourth phase is a warm-up. The runtime captures CUDA graphs for the common request shapes and runs at least one forward pass so the first real request does not pay JIT or graph-capture overhead. This adds a few seconds. The two wrong options describe work that happens before any of this: model code is built ahead of time, and tokenizers are trained once and shipped as artifacts. Neither runs at cold start.

**Cold start is four phases, each with different magnitudes.** First, pod scheduling and container image pull. The orchestrator picks a node, the node pulls the runtime image, the container starts. Tens of seconds, depending on image size and registry bandwidth. Second, the weight download. The model lives in object storage; the replica streams it to local disk. For a 70B-class model that is tens of gigabytes and minutes of transfer. This is the dominant cost in almost every production setup. Third, the load from disk into HBM. Once the bytes are on local NVMe, copying them into device memory is bandwidth-limited and runs in the seconds range. Fourth, the warm-up. CUDA graph capture for the common request shapes plus at least one forward pass before traffic is admitted. A few seconds, but skipping it makes the first real request pay JIT and capture overhead, which is a visible UX hit. The two distractors invent steps that do not exist. Transformer code ships as pre-built kernels and binaries in the runtime image; nothing is recompiled from source on cold start. The tokenizer is a fixed artifact trained once and shipped with the model; it is never retrained per replica. The mitigation playbook follows directly: pre-bake images with the runtime, cache weights on local NVMe across pod restarts where possible, keep a warm pool of pre-loaded replicas to absorb traffic spikes, and document the cold-start budget so the autoscaler knows how aggressively to over-provision.

Quick reference

TL;DR

Real cold-start cost is pod scheduling plus image pull, weight download to disk, weight load to HBM, and a warm-up forward pass. Recompiling from source or retraining the tokenizer are made-up distractors.

Memory aid

Easy to grasp

Imagine opening a new branch of a restaurant first thing in the morning. The building has to be ready, the supplies have to be trucked in, the chef has to unpack everything into the kitchen, and the chef has to cook one practice meal before the doors open. Each of those is a real step that takes real time. What does not happen is the chef inventing the recipes from scratch on opening morning, or training the suppliers to grow the vegetables. The recipes and the suppliers are already settled. For an LLM replica, the building is the pod, the supplies are the model's learned numbers, and the practice meal is one warm-up run before customers arrive. Spotting the distractors is just spotting the steps that belong to building the restaurant chain, not to opening today's branch.

Key concepts

Cold start Autoscaling Model serving

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: four phases of cold start + which one dominates for large models + warm-up cost + why the distractors fail + mitigation playbook + autoscaling implications.

Real-world examples

Real products, models, and research that use this idea.

vLLM and SGLang both document cold-start guidance recommending pre-staged weights on local NVMe to cut the dominant download phase.
Hugging Face Text Generation Inference exposes a warm-up endpoint that callers hit before admitting traffic to a fresh replica.
AWS SageMaker and GCP Vertex AI maintain warm pools of pre-loaded inference endpoints precisely because reactive cold-start cannot meet interactive SLOs.
NVIDIA TensorRT-LLM ships pre-built engine plans so deployment does not pay a kernel-compilation cost at cold start.
Anthropic, OpenAI, and DeepSeek production fleets run capacity ahead of demand based on prewarm forecasts, which is how interactive products survive traffic spikes.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you reduce weight-download time without changing the model?

QWhy does CUDA graph capture matter even when the GPU has the weights?

QHow does cold start interact with the autoscaler's reaction time?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating cold start as one number. It is four phases with very different magnitudes, and the dominant one is almost always weight download to local disk, not CUDA graph capture.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.