Spot the error: 'we autoscale our LLM serving pods on CPU utilization.'

Spot the error·Medium·4.0 · 0·~2 min·Asked atInfosys Tcs Writer·Relevant atMicrosoft

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

An LLM serving pod runs on a GPU node. The CPU on that pod has very little to do: it accepts the HTTP request, runs the tokenizer, hands the tokens to the engine, and ships bytes back. All the heavy work, the prefill and decode passes, happens on the GPU's tensor cores against memory in HBM. Under real load the CPU usually sits between 5 and 20 percent. Meanwhile the GPU can be pinned at 100 percent with a request queue piling up behind it. If the autoscaler is watching CPU, it sees a calm number and does nothing, while users on the GPU side see time-to-first-token spike from 200 ms to 5 seconds. The autoscaler needs to watch the part of the system that actually saturates. Good signals include the queue of waiting requests, the in-flight batch size relative to KV-cache capacity, GPU SM utilization, or pending-token count from the engine. Kubernetes does not expose these by default, but KEDA (Kubernetes Event-Driven Autoscaling) bridges the gap. The vLLM and SGLang scheduler metrics endpoints publish queue depth and batch fullness directly; you point KEDA at them and the HPA scales on the right signal. The error in the design doc is choosing the convenient signal over the correct one.

The autoscaler is reading the wrong meter. LLM serving on an H100 is GPU-bound: prefill and decode hammer the tensor cores and HBM, while the CPU is doing tokenization, scheduling, and request marshalling. Under sustained overload CPU might reach 15 percent on a pod whose GPU is at 100 percent with dozens of requests queued. This breaks the autoscaling loop. The HPA's job is to add replicas BEFORE users feel pain. CPU at 70 percent only happens on an LLM pod when something is catastrophically wrong (think GIL contention from a single Python process serving thousands of small requests). By then the GPU has been pinned for minutes and TTFT has blown the SLO budget. The right signals all track what is actually saturating: - **Queue depth**: requests waiting to be admitted into a batch. Rises immediately when the GPU cannot keep up. - **In-flight batch size vs KV-cache capacity**: tells you the GPU is full of conversation state and cannot admit more. - **GPU SM utilization**: NVIDIA-DCGM exposes this; high sustained values plus rising queue mean scale up. - **Pending-token count**: total tokens enqueued across all sessions; correlates with TTFT degradation. KEDA wires Kubernetes HPA to these signals via Prometheus scaling triggers. vLLM, SGLang, and TensorRT-LLM all expose them on their metrics endpoints. The fix is one config change, not a redesign.

TL;DR

CPU utilisation barely moves on an LLM serving pod because the work is on the GPU; autoscale on queue depth, batch fullness, GPU SM-util, or KV-block occupancy instead.

Memory aid

Easy to grasp

Picture a restaurant where the kitchen does all the cooking but the autoscaler watches how busy the maitre d' is. The maitre d' just greets people and writes orders; he is never overwhelmed even when the kitchen is on fire and tickets are piling up behind the line. By the time the maitre d' looks busy, the kitchen has been buried for an hour. LLM serving works the same way. The CPU is the maitre d', taking requests and writing tickets. The GPU is the kitchen, doing the actual cooking. The autoscaler should watch what is going on in the kitchen: how long the ticket queue is, how full each pan, how hot the stove. Watching the maitre d' looks reasonable on paper and misses every real problem.

Key concepts

Autoscaling Horizontal Pod Autoscaler GPU utilization KV cache

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Autoscaling LLM serving is one of the most consequential infrastructure decisions a serving team makes, and the design doc in this question gets it wrong in a way that is easy to miss because the choice looks reasonable. The HPA supports CPU out of the box; CPU is the lingua franca of Kubernetes autoscaling; it has worked for every web service the team has ever run. So why not LLMs?

The answer is that LLM serving sits in a completely different bottleneck regime than traditional web services. A request to a CRUD API spends most of its life on the CPU: deserialising JSON, running business logic, hitting a database, serialising a response. Scaling on CPU works because CPU saturation IS the operational pain. A request to an LLM serving pod spends almost none of its life on the CPU. Tokenisation is fast; HTTP framing is fast; everything else is GPU. Scaling on CPU therefore measures the wrong meter, and the wrong meter never crosses the threshold no matter how badly the GPU is overloaded.

This deep dive walks through the bottleneck anatomy of an LLM serving pod, lays out the menu of correct signals and what each one measures, shows how to wire them into a Kubernetes HPA via KEDA, covers the scale-up versus scale-down asymmetry that production deployments need, and addresses the cold-start problem that affects how aggressively you can scale. By the end the goal is to be able to read any autoscaling design doc and immediately spot the same category error elsewhere.

Anatomy of work on an LLM serving pod

An incoming request to a serving pod traverses these stages:

HTTP intake, JSON parse, route. A few microseconds on the CPU.
Tokenisation. A few milliseconds on the CPU for typical prompts; very fast on Rust-backed tokenisers like tokenizers-rs.
Admission to the scheduler queue. Negligible CPU.
Prefill on the GPU. Tens to hundreds of milliseconds depending on prompt length; tensor-core bound on H100.
Decode on the GPU. One forward pass per output token, bandwidth-bound on HBM.
Detokenisation and SSE streaming. A few microseconds per token on the CPU.

Across the whole request lifecycle the CPU is busy for a few percent of wall time at most. Even at very high request rates per pod (hundreds of concurrent streams) the CPU side parallelises trivially across a few worker threads and rarely crosses 20 percent on a typical 64-vCPU node.

The GPU side, by contrast, hits capacity quickly. KV cache fills with in-flight sessions; tensor cores or HBM saturate at production batch sizes. Once the GPU is full, new requests queue at the scheduler. The queue is where overload first becomes visible.

This asymmetry is what makes CPU the wrong autoscaling signal. The signal you choose has to climb in lockstep with the actual saturation. CPU does not.

The menu of correct signals

Wiring it through KEDA and DCGM

Scale-up versus scale-down asymmetry

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM exposes `num_requests_waiting`, `num_requests_running`, and `gpu_cache_usage_perc` on its Prometheus endpoint; these are the canonical scale-up signals.
SGLang publishes `pending_requests` and `running_requests` for the same purpose, with KEDA recipes documented in 2026.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does scaling on queue depth alone tend to flap under bursty traffic?

Queue depth spikes instantly on a traffic burst, triggering scale-up, then drains quickly once the burst passes, triggering scale-down. The cycle of scale-up, scale-down, scale-up creates churn. Production stacks usually scale up fast on queue depth above a high threshold but scale down on a moving-window average of utilisation, asymmetric to avoid flapping.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating CPU as a reasonable proxy because the HPA supports it out of the box. The default signal is not the right signal: it never crosses the threshold during real overload, so the autoscaler reacts after the SLO has already broken.

60 second bullets to scan on the way to the call.

Why CPU stays low on an LLM serving pod even under GPU overload
Which scheduler metrics vLLM and SGLang expose for autoscaling

Primary sources. Browse if you want the original framing.