Why a self-hosted LLM deployment needs a warm up step before traffic and how it differs from cache priming

Flashcard·Easy·4.0 · 0·~30s·Asked atCognizant Deloitte Observe Ai

Attempt it

The pod has to load weights into GPU memory (tens of seconds for a 70B model). vLLM also compiles the model, captures CUDA graphs for a set of batch sizes and pre-allocates the KV-cache block pool at startup, before it serves any request. A real request routed to a pod that is still starting fails or times out under load.

Warm-up sends representative shapes (a couple of varying input lengths, the same kinds of output the workload produces) once the server is up. It runs the model end to end, triggers any remaining lazy initialization and confirms the pod serves at normal latency before traffic arrives. It is not the same as prompt cache priming, which warms the provider's KV cache for a frequently-used system prompt prefix; warm-up is about the local pod's state. Both matter; both should run before the load balancer puts the pod into rotation.

Your answer

When a new **vLLM** pod starts, three things happen before it can serve traffic. First, the model weights load into GPU memory; for a 70B parameter model at FP8 this is tens of seconds of disk to GPU transfer. Second, the runtime compiles its **CUDA kernels** and captures CUDA graphs for a set of batch sizes. Third, it pre-allocates the **PagedAttention** KV-cache block pool. vLLM finishes all of this before it serves any request. If a load balancer routes real traffic to a fresh pod at once, requests reach a pod that is still starting. They fail or time out, the load balancer marks the pod unhealthy, and the autoscaler thinks the pod failed. Under load this cascades. **Warm-up** fires a few synthetic requests at the pod (with representative input lengths and output shapes) before the load balancer puts it into rotation. They run the model end to end, trigger any remaining lazy initialization, and confirm the pod serves at production latency. **Warm-up is not the same as prompt cache priming.** Prompt cache priming targets the provider's KV cache for a frequently-used system prompt prefix; warm-up targets the local pod's state. Both happen before traffic; they address different layers.

**Three costs on a cold pod, paid at startup.** **Weight loading.** Model weights move from disk or a model-registry blob store into GPU memory. For Llama 3.3 70B at FP8 this is roughly 70GB of data and tens of seconds even on fast storage; for a 405B-class model it can be minutes. Until weights are loaded, the pod cannot serve any request. **Kernel compilation.** vLLM compiles the model and captures CUDA graphs for a list of batch sizes. All compilation finishes before the server takes any request. **KV-cache allocation.** vLLM pre-allocates the PagedAttention block pool from a set share of GPU memory (`gpu_memory_utilization`), so the pool exists before the first request. **Warm-up fires synthetic requests with representative shapes once startup ends.** A reasonable routine sends two or three requests covering the input-length distribution (a short, a medium, a long), each generating a representative output length. This runs the model end to end so any remaining lazy initialization happens before real users see it. It also confirms that each shape serves at the expected latency. The pod only joins the load-balancer pool after warm-up completes. **Prompt cache priming is a separate, complementary concern.** Anthropic's prompt caching, OpenAI's prompt caching, and similar mechanisms maintain a KV cache for a stable system-prompt prefix on the provider's side. Priming this cache means sending a request that includes the full system prompt early in the deployment so subsequent requests with the same prefix hit the cache and pay only the suffix-recompute cost.

TL;DR

Startup loads weights, compiles kernels and pre-allocates the KV cache; warm-up requests then test the full path before traffic. Prompt cache priming warms the provider's KV cache, a different layer. Both are needed.

Memory aid

Easy to grasp

Imagine a brand-new bakery oven on opening day. Picture pulling a loaf out at the right time when the oven was room-temperature when you started. The bread comes out raw, the line of customers backs up, and the owner blames the recipe. Getting an LLM pod ready is the same preheat. Load the weights, get the trays in place, allocate the rack space. Cache priming is a different thing. Think of it as making sure the dough you reach for most often is already proofed on the counter, not still frozen in the back. Both matter. Both happen before the first customer order. Skip either and the first hour of service is ugly.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Cold pods are the single most common cause of bad first-deploy outcomes for self-hosted LLM stacks. A team that has not encountered warm-up before will deploy a new pod, route traffic to it, watch the first batch of user requests time out, see the load balancer mark the pod unhealthy, and conclude that the runtime is broken. The runtime is fine; the deploy missed a step.

Three layers of cold-pod state

Layer 1, Weights in GPU memory. When a vLLM pod starts, the model weights have to load from disk (or a network blob store) into GPU memory. For a Llama 3.3 70B at FP8 this is roughly 70GB of data; transfer times are tens of seconds on fast local SSD, longer on network-attached storage. For 405B-class models the transfer can take minutes. Until weights are loaded the pod cannot serve any request, period.

The mitigations are infrastructure-side: pin the model to a node with fast local storage, use parallel sharded loading across multiple GPUs (vLLM supports this for tensor-parallel deployments), and avoid network-attached storage for the model directory unless the read bandwidth has been measured and is sufficient. A model-registry cache (Hugging Face Hub mirror, or a private blob store with prefetching) can shave additional time off cold starts.

Layer 2, Compiled CUDA kernels and graphs. vLLM uses specialized CUDA kernels (PagedAttention, fused MLPs, sampling). At startup it compiles the model and captures CUDA graphs for a list of batch sizes; later batches are padded to a captured size. All compilation finishes before the server serves any request, so this cost lands in pod start time rather than on the first user requests.

Layer 3, PagedAttention block pool. vLLM pre-allocates the physical KV block pool on the GPU at startup, from a set share of GPU memory (gpu_memory_utilization). The pool exists before the first request, and sequences pull blocks from the free list at near-zero cost. This layer is the cheapest of the three.

What warm-up actually does

Warm-up vs prompt cache priming

Readiness probes and the serialization discipline

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama 3.3 70B served on vLLM with a warm-up script that sends three synthetic requests covering short, medium, and long input lengths before the readiness probe passes.
SGLang deployment with a Kubernetes readiness probe that fails until the warm-up endpoint returns success; the LB only routes after warm-up.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design the warm-up script for a workload that has very long input contexts (32K-128K tokens) and unpredictable output lengths?

Sample a few representative input lengths from the actual distribution (not synthetic uniform samples), for example one short, one mid, one near-max. For each, generate a representative output length that exercises the long-generation path. Cap the warm-up budget at a few seconds total; you do not need exhaustive coverage. The goal is exercising the hot path end to end, not warming every possible shape. Verify warm-up effectiveness by measuring p99 time to first token on the first 10 production requests; if it matches steady-state, warm-up is sufficient.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Conflating warm-up with prompt cache priming. Warm-up is about the local pod's state; prompt cache priming is about the provider's KV cache for a stable prefix. Different layers, different mitigations.

60 second bullets to scan on the way to the call.

The three cold-pod costs: weight loading, kernel compilation, KV-allocator initialization
Why warm-up sends representative input lengths rather than one canonical shape

Primary sources. Browse if you want the original framing.