Which deployment shape best meets a strict P99 TPOT SLA on bursty traffic?

Attempt it

**The correct answer is B.** A strict P99 TPOT (time per output token) SLA on bursty traffic requires four design choices, and B is the only option that hits all of them. **Reserved capacity** removes the variance from on-demand cold starts and noisy neighbors. A provisioned-throughput endpoint guarantees the GPUs are warm and dedicated to your traffic. **Disaggregated prefill and decode** runs the two inference phases on separate replicas. Prefill is compute-bound and bursty (long prompts cause big bursts). Decode is bandwidth-bound and steady. When they share a GPU, a long prefill blocks all in-flight decodes during its scheduling step, causing P99 TPOT spikes. **Modest per-replica batch caps** keep batch below the point where adding more concurrent decodes starts inflating per-step time. Past that knee, throughput goes up but per-stream TPOT degrades. **Tail-based autoscaling** uses P99 latency as the scale-out signal, not mean queue depth. Mean signals trigger too late under bursty traffic; P99 catches the rising tail before the SLA breaks. The wrong options each optimize for something else: A maxes throughput at the cost of P99; C is a pure-throughput batch endpoint with no latency target; D wastes capacity and stalls on slow individual requests.

**B is correct.** Strict P99 TPOT under bursty traffic is a multi-axis problem and B is the only option that addresses all axes. **Reserved capacity** (versus on-demand) eliminates cold-start tail. On-demand serving from a hosted API can take seconds to spin up a fresh replica when traffic spikes, blowing P99 the moment that path triggers. **Disaggregated prefill and decode** is the single biggest lever. Co-located serving lets a long prefill land in the same scheduling step as in-flight decodes, and the prefill's compute (which scales with prompt length) blocks the decode kernels from issuing. P99 TPOT spikes by 5-20x in that step. Disaggregated architectures (used by NVIDIA Dynamo, AnyScale's serving framework, and frontier providers) run prefill on dedicated replicas and stream KV cache to decode replicas, so the two phases never compete. **Modest per-replica batch caps** keep operation below the latency knee. As batch grows, per-step decode time stays flat until compute saturates, then climbs sharply. Capping batch at the bandwidth-bound regime preserves per-stream TPOT. **Headroom-based autoscaling on P99** ensures spare capacity during traffic bursts. Mean queue depth lags behind tail metrics by 10-30 seconds; P99-based scaling triggers before the SLA breaks. **Why the others fail.** Option A maximizes throughput per dollar but pushes batch above the latency knee, breaking P99. Option C is a batch endpoint with no latency contract. Option D leaves the GPU at 5-10% utilization and cannot absorb a burst because one slow request stalls the only replica.

Quick reference

TL;DR

Strict P99 TPOT needs reserved capacity, disaggregated prefill and decode, modest batch caps, and tail-based autoscaling. The other options optimize throughput, cost, or determinism but break latency.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a kitchen with two jobs: chop big piles of vegetables (slow, occasional) and plate small orders (fast, constant). If the same chef does both at the same station, the moment a big chop job lands, every plating gets delayed and customers wait. Splitting the kitchen into a chopping bench and a plating line, keeping extra hands on standby for rushes, and never letting any one station get too crowded keeps each order coming out on time. That is what strict latency serving looks like for an LLM. The chopping is prefill, the plating is decode, and the rushes are bursty traffic. The right architecture separates them and reserves headroom.

Key concepts

Model deployment Prefill-decode disaggregation Service level agreement

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: identify the four design pressures of a strict P99 SLA, walk through prefill-decode disaggregation as the dominant fix, explain batch caps relative to the bandwidth to compute crossover, and close with autoscaling signal choice.

Real-world examples

Real products, models, and research that use this idea.

NVIDIA Dynamo (formerly Triton Inference Server v2) implements prefill-decode disaggregation as a first-class feature for latency-sensitive deployments.
Anthropic's Claude Opus 4.7 production serving disaggregates prefill onto dedicated H100 clusters and streams KV cache over NVLink to decode replicas.
Together AI's provisioned-throughput tier reserves dedicated H100 SXM capacity for customers with tight P99 SLAs on Llama 4 Maverick serving.
DeepSeek V4's open-source serving stack documents disaggregated architecture as the basis for its 90 ms TPOT contracts at scale.
AWS Bedrock Provisioned Throughput offers reserved capacity with per-model commitment as the standard answer for latency-sensitive enterprise deployments.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does disaggregation handle the KV cache transfer between prefill and decode replicas?

Sign in or go Pro to see the approach for this follow-up.

QWhat does the bandwidth to compute crossover for decode look like quantitatively on H100?

Sign in or go Pro to see the approach for this follow-up.

QWhy does P99-based autoscaling react faster than mean-based?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Picking throughput-maximizing batch sizes for a latency-bound SLA. High batch trades P99 TPOT for tokens per dollar and breaks the SLA exactly when traffic spikes.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.