Which components dominate cold start latency when a serving replica spins up?

Multi-select·Medium·4.0 · 0·~1 min·Asked atDatabricks Microsoft Robinhood·Relevant atCloudflare Groq Modal Labs

Attempt it

**Cold start** is the wall-clock time between an autoscaler deciding to add a replica and that replica being ready to serve its first request. For LLM serving it is usually measured in minutes, not seconds, and it walks through four phases that all happen every time. The first phase is pod scheduling and container image pull. Kubernetes (or whatever orchestrator) picks a node, the node pulls the serving runtime container from the registry, and the kubelet starts the container. This takes tens of seconds depending on image size and registry bandwidth. The second phase is the model-weight download. Weights live in object storage like S3 or GCS, and the replica streams them to its local disk on startup. For a 70B-class model this is tens of gigabytes and can take minutes. This phase is the usual bottleneck. The third phase loads the on-disk weights into GPU HBM. Once the bytes are on the local NVMe, copying them into device memory is a few seconds. The fourth phase is a warm-up. The runtime captures CUDA graphs for the common request shapes and runs at least one forward pass so the first real request does not pay JIT or graph-capture overhead. This adds a few seconds. The two wrong options describe work that happens before any of this: model code is built ahead of time, and tokenizers are trained once and shipped as artifacts. Neither runs at cold start.

**Cold start is four phases, each with different magnitudes.** First, pod scheduling and container image pull. The orchestrator picks a node, the node pulls the runtime image, the container starts. Tens of seconds, depending on image size and registry bandwidth. Second, the weight download. The model lives in object storage; the replica streams it to local disk. For a 70B-class model that is tens of gigabytes and minutes of transfer. This is the dominant cost in almost every production setup. Third, the load from disk into HBM. Once the bytes are on local NVMe, copying them into device memory is bandwidth-limited and runs in the seconds range. Fourth, the warm-up. CUDA graph capture for the common request shapes plus at least one forward pass before traffic is admitted. A few seconds, but skipping it makes the first real request pay JIT and capture overhead, which is a visible UX hit. The two distractors invent steps that do not exist. Transformer code ships as pre-built kernels and binaries in the runtime image; nothing is recompiled from source on cold start. The tokenizer is a fixed artifact trained once and shipped with the model; it is never retrained per replica. The mitigation playbook follows directly: pre-bake images with the runtime, cache weights on local NVMe across pod restarts where possible, keep a warm pool of pre-loaded replicas to absorb traffic spikes, and document the cold-start budget so the autoscaler knows how aggressively to over-provision.

TL;DR

Real cold-start cost is pod scheduling plus image pull, weight download to disk, weight load to HBM, and a warm-up forward pass. Recompiling from source or retraining the tokenizer are made-up distractors.

Memory aid

Easy to grasp

Imagine opening a new branch of a restaurant first thing in the morning. The building has to be ready, the supplies have to be trucked in, the chef has to unpack everything into the kitchen, and the chef has to cook one practice meal before the doors open. Each of those is a real step that takes real time. What does not happen is the chef inventing the recipes from scratch on opening morning, or training the suppliers to grow the vegetables. The recipes and the suppliers are already settled. For an LLM replica, the building is the pod, the supplies are the model's learned numbers, and the practice meal is one warm-up run before customers arrive. Spotting the distractors is just spotting the steps that belong to building the restaurant chain, not to opening today's branch.

Key concepts

Cold start Autoscaling Model serving

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Cold start is the wall-clock cost of bringing a serving replica from nothing to ready-to-accept-traffic. For LLM workloads it is a minutes-scale problem, not a seconds-scale problem, and that magnitude shapes the entire autoscaling and capacity architecture above it. A team that does not understand cold start will routinely ship reactive autoscaling and discover, on the first real traffic spike, that the replicas arrive minutes after they were needed.

This question tests whether you can decompose cold start into its real phases and discriminate them from plausible-but-fictional steps. The four real phases (pod scheduling and image pull, weight download to local disk, load from disk into HBM, warm-up forward pass) are not equally costly. Knowing which one dominates is what makes the mitigation playbook concrete.

The deep dive walks each phase, shows why two listed options are distractors, and connects cold-start budget to the autoscaling policy that has to live with it. By the end you should be able to estimate cold-start time for a model and hardware combination and design the warm-pool sizing that protects the SLO during traffic spikes.

Phase one: pod scheduling and container image pull

The orchestrator (Kubernetes, Nomad, ECS) decides which node hosts the new replica, then the kubelet on that node pulls the container image from the registry. The image holds the serving runtime (vLLM, SGLang, TRT-LLM), its dependencies, and usually a base CUDA layer.

Image pull time depends on image size and registry-to-node bandwidth. A typical serving image is several GB; pulling at hundreds of MB/s puts this phase in the tens-of-seconds range. Nodes that already have the image cached skip this phase entirely, which is the case for pre-baked node pools and for replicas restarted on a node that recently ran the same image.

The mitigation is straightforward: pre-bake the node AMI or VM image with the serving runtime preloaded, use registry caching at the regional or node level, and keep image sizes lean so the pull is fast when it does happen. This phase is real but rarely the bottleneck.

Phase two: weight download from object storage

Phase three: load from disk into HBM

Phase four: warm-up and CUDA graph capture

Why the two distractors fail

From cold-start budget to autoscaling policy

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM and SGLang both document cold-start guidance recommending pre-staged weights on local NVMe to cut the dominant download phase.
Hugging Face Text Generation Inference exposes a warm-up endpoint that callers hit before admitting traffic to a fresh replica.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you reduce weight-download time without changing the model?

Pre-stage weights onto local NVMe via a sidecar that runs ahead of the serving container, mount a regional cache layer in front of object storage, or use a peer-to-peer pre-fetch across the fleet. Each trades a bit of infra complexity for minutes of cold-start time.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating cold start as one number. It is four phases with very different magnitudes, and the dominant one is almost always weight download to local disk, not CUDA graph capture.

60 second bullets to scan on the way to the call.

The four phases of cold start and their relative magnitudes
Why weight download from object storage is the dominant phase for large models

Primary sources. Browse if you want the original framing.