Why a self-hosted LLM deployment needs a warm up step before traffic and how it differs from cache priming
Warm-up materializes weights, compiled CUDA kernels, and the KV-page allocator before traffic; prompt cache priming warms the provider's KV cache for a hot system prompt, different layers, both needed.
Imagine a brand-new bakery oven on opening day. Picture pulling a loaf out at the right time when the oven was room-temperature when you started. The bread comes out raw, the line of customers backs up, and the owner blames the recipe. Warm-up for an LLM pod is the same preheat. Load the weights, get the trays in place, allocate the rack space. Cache priming is a different thing. Think of it as making sure the dough you reach for most often is already proofed on the counter, not still frozen in the back. Both matter. Both happen before the first customer order. Skip either and the first hour of service is ugly.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Cold pods are the single most common cause of bad first-deploy outcomes for self-hosted LLM stacks. A team that has not encountered warm-up before will deploy a new pod, route traffic to it, watch the first batch of user requests time out, see the load balancer mark the pod unhealthy, and conclude that the runtime is broken. The runtime is fine; the deploy missed a step.
This walkthrough explains the three layers of cold-pod state that warm-up addresses, distinguishes warm-up from the often-confused prompt cache priming, and describes the readiness-probe discipline that makes both work in production.
Three layers of cold-pod state
Layer 1, Weights in GPU memory. When a vLLM or TGI pod starts, the model weights have to load from disk (or a network blob store) into GPU memory. For a Llama 4 70B at FP8 this is roughly 70GB of data; transfer times are tens of seconds on fast local SSD, longer on network-attached storage. For 405B-class models the transfer can take minutes. Until weights are loaded the pod cannot serve any request, period.
The mitigations are infrastructure-side: pin the model to a node with fast local storage, use parallel sharded loading across multiple GPUs (both vLLM and TGI support this for tensor-parallel deployments), and avoid network-attached storage for the model directory unless the read bandwidth has been measured and is genuinely sufficient. A model-registry cache (Hugging Face Hub mirror, or a private blob store with prefetching) can shave additional time off cold starts.
Layer 2, Compiled CUDA kernels for batch shapes. vLLM and TGI use specialized CUDA kernels (PagedAttention, fused MLPs, sampling) that are JIT-compiled the first time a particular batch shape (sequence length, batch size combination) is seen. The first request with a previously-unseen shape pays the compilation cost, hundreds of milliseconds to a few seconds depending on the shape and the GPU. Under bursty load with varied input lengths, a pod that has not been warmed can pay these costs repeatedly for the first few minutes of life.
Layer 3, PagedAttention page pool initialization. The KV-cache allocator has to allocate the physical block pool on the GPU and initialize the free-list data structures. The first sequence that grows the pool incurs allocation cost; once the pool is established, subsequent sequences pull from the free list at near-zero cost. This layer is the cheapest of the three but is exercised on the first generation of any meaningful length.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 4 70B served on vLLM with a warm-up script that sends three synthetic requests covering short, medium, and long input lengths before the readiness probe passes.
- TGI deployment with a Kubernetes readiness probe that fails until the warm-up endpoint returns success; the LB only routes after warm-up.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design the warm-up script for a workload that has very long input contexts (32K-128K tokens) and unpredictable output lengths?
Sample a few representative input lengths from the actual distribution (not synthetic uniform samples), for example one short, one mid, one near-max. For each, generate a representative output length that exercises the long-generation path so the KV-cache page pool is warmed for that pattern. Cap the warm-up budget at a few seconds total; you do not need exhaustive coverage. The goal is materializing the hot-path kernels, not warming every possible shape. Verify warm-up effectiveness by measuring p99 time to first token on the first 10 production requests; if it matches steady-state, warm-up is sufficient.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Conflating warm-up with prompt cache priming. Warm-up is about the local pod's compiled state and allocator; prompt cache priming is about the provider's KV cache for a stable prefix. Different layers, different mitigations.
60 second bullets to scan on the way to the call.
The three cold-pod costs: weight loading, kernel compilation, KV-allocator initialization
Why warm-up sends representative input lengths rather than one canonical shape
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.