Spot the error: 'we autoscale our LLM serving pods on CPU utilization.'

Spot the error·Medium·4.0 · 0·~2 min·Relevant atMicrosoft

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

An LLM serving pod runs on a GPU node. The CPU on that pod has very little to do: it accepts the HTTP request, runs the tokenizer, hands the tokens to the engine, and ships bytes back. All the heavy work, the prefill and decode passes, happens on the GPU's tensor cores against memory in HBM. Under real load the CPU usually sits between 5 and 20 percent. Meanwhile the GPU can be pinned at 100 percent with a request queue piling up behind it. If the autoscaler is watching CPU, it sees a calm number and does nothing, while users on the GPU side see time-to-first-token spike from 200 ms to 5 seconds. The autoscaler needs to watch the part of the system that actually saturates. Good signals include the queue of waiting requests, the in-flight batch size relative to KV-cache capacity, GPU SM utilization, or pending-token count from the engine. Kubernetes does not expose these by default, but KEDA (Kubernetes Event-Driven Autoscaling) bridges the gap. The vLLM and SGLang scheduler metrics endpoints publish queue depth and batch fullness directly; you point KEDA at them and the HPA scales on the right signal. The error in the design doc is choosing the convenient signal over the correct one.

The autoscaler is reading the wrong meter. LLM serving on an H100 is GPU-bound: prefill and decode hammer the tensor cores and HBM, while the CPU is doing tokenization, scheduling, and request marshalling. Under sustained overload CPU might reach 15 percent on a pod whose GPU is at 100 percent with dozens of requests queued. This breaks the autoscaling loop. The HPA's job is to add replicas BEFORE users feel pain. CPU at 70 percent only happens on an LLM pod when something is catastrophically wrong (think GIL contention from a single Python process serving thousands of small requests). By then the GPU has been pinned for minutes and TTFT has blown the SLO budget. The right signals all track what is actually saturating: - **Queue depth**: requests waiting to be admitted into a batch. Rises immediately when the GPU cannot keep up. - **In-flight batch size vs KV-cache capacity**: tells you the GPU is full of conversation state and cannot admit more. - **GPU SM utilization**: NVIDIA-DCGM exposes this; high sustained values plus rising queue mean scale up. - **Pending-token count**: total tokens enqueued across all sessions; correlates with TTFT degradation. KEDA wires Kubernetes HPA to these signals via Prometheus scaling triggers. vLLM, SGLang, and TensorRT-LLM all expose them on their metrics endpoints. The fix is one config change, not a redesign.

Quick reference

TL;DR

CPU utilisation barely moves on an LLM serving pod because the work is on the GPU; autoscale on queue depth, batch fullness, GPU SM-util, or KV-block occupancy instead.

Memory aid

Easy to grasp

Picture a restaurant where the kitchen does all the cooking but the autoscaler watches how busy the maitre d' is. The maitre d' just greets people and writes orders; he is never overwhelmed even when the kitchen is on fire and tickets are piling up behind the line. By the time the maitre d' looks busy, the kitchen has been buried for an hour. LLM serving works the same way. The CPU is the maitre d', taking requests and writing tickets. The GPU is the kitchen, doing the actual cooking. The autoscaler should watch what is going on in the kitchen: how long the ticket queue is, how full each pan, how hot the stove. Watching the maitre d' looks reasonable on paper and misses every real problem.

Key concepts

Autoscaling Horizontal Pod Autoscaler GPU utilization KV cache

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: why CPU does not saturate on a GPU-bound pod + the right signal set (queue, batch, KV, SM-util) + KEDA + DCGM wiring + scale-up vs scale-down asymmetry + cold-start considerations.

Real-world examples

Real products, models, and research that use this idea.

vLLM exposes `num_requests_waiting`, `num_requests_running`, and `gpu_cache_usage_perc` on its Prometheus endpoint; these are the canonical scale-up signals.
SGLang publishes `pending_requests` and `running_requests` for the same purpose, with KEDA recipes documented in 2026.
NVIDIA DCGM exporter has been the standard way to surface GPU SM utilization to Prometheus and Kubernetes HPA since 2023.
KServe and Ray Serve both ship GPU-aware autoscalers that use batch utilisation rather than CPU as the primary signal.
AWS Bedrock provisioned throughput exposes a tokens-per-minute metric for autoscaling; CPU is not even an option in the API.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does scaling on queue depth alone tend to flap under bursty traffic?

QHow does KV-cache occupancy compare to GPU SM utilisation as a scale-up signal?

QWhat is the right way to scale a model that needs minutes to load weights into the GPU?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating CPU as a reasonable proxy because the HPA supports it out of the box. The default signal is not the right signal: it never crosses the threshold during real overload, so the autoscaler reacts after the SLO has already broken.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.