Which deployment shape best meets a strict P99 TPOT SLA on bursty traffic?

MCQ·Medium·4.0 · 0·~1 min·Asked atCapgemini Microsoft Sharechat·Relevant atDatabricks

Attempt it

**The correct answer is B.** A strict P99 TPOT (time per output token) SLA on bursty traffic requires four design choices, and B is the only option that hits all of them. **Reserved capacity** removes the variance from on-demand cold starts and noisy neighbors. A provisioned-throughput endpoint guarantees the GPUs are warm and dedicated to your traffic. **Disaggregated prefill and decode** runs the two inference phases on separate replicas. Prefill is compute-bound and bursty (long prompts cause big bursts). Decode is bandwidth-bound and steady. When they share a GPU, a long prefill blocks all in-flight decodes during its scheduling step, causing P99 TPOT spikes. **Modest per-replica batch caps** keep batch below the point where adding more concurrent decodes starts inflating per-step time. Past that knee, throughput goes up but per-stream TPOT degrades. **Tail-based autoscaling** uses P99 latency as the scale-out signal, not mean queue depth. Mean signals trigger too late under bursty traffic; P99 catches the rising tail before the SLA breaks. The wrong options each optimize for something else: A maxes throughput at the cost of P99; C is a pure-throughput batch endpoint with no latency target; D wastes capacity and stalls on slow individual requests.

**B is correct.** Strict P99 TPOT under bursty traffic is a multi-axis problem and B is the only option that addresses all axes. **Reserved capacity** (versus on-demand) eliminates cold-start tail. On-demand serving from a hosted API can take seconds to spin up a fresh replica when traffic spikes, blowing P99 the moment that path triggers. **Disaggregated prefill and decode** is the single biggest lever. Co-located serving lets a long prefill land in the same scheduling step as in-flight decodes, and the prefill's compute (which scales with prompt length) blocks the decode kernels from issuing. P99 TPOT spikes by 5-20x in that step. Disaggregated architectures (used by NVIDIA Dynamo, AnyScale's serving framework, and frontier providers) run prefill on dedicated replicas and stream KV cache to decode replicas, so the two phases never compete. **Modest per-replica batch caps** keep operation below the latency knee. As batch grows, per-step decode time stays flat until compute saturates, then climbs sharply. Capping batch at the bandwidth-bound regime preserves per-stream TPOT. **Headroom-based autoscaling on P99** ensures spare capacity during traffic bursts. Mean queue depth lags behind tail metrics by 10-30 seconds; P99-based scaling triggers before the SLA breaks. **Why the others fail.** Option A maximizes throughput per dollar but pushes batch above the latency knee, breaking P99. Option C is a batch endpoint with no latency contract. Option D leaves the GPU at 5-10% utilization and cannot absorb a burst because one slow request stalls the only replica.

TL;DR

Strict P99 TPOT needs reserved capacity, disaggregated prefill and decode, modest batch caps, and tail-based autoscaling. The other options optimize throughput, cost, or determinism but break latency.

Memory aid

Easy to grasp

Imagine a kitchen with two jobs: chop big piles of vegetables (slow, occasional) and plate small orders (fast, constant). If the same chef does both at the same station, the moment a big chop job lands, every plating gets delayed and customers wait. Splitting the kitchen into a chopping bench and a plating line, keeping extra hands on standby for rushes, and never letting any one station get too crowded keeps each order coming out on time. That is what strict latency serving looks like for an LLM. The chopping is prefill, the plating is decode, and the rushes are bursty traffic. The right architecture separates them and reserves headroom.

Key concepts

Model deployment Prefill-decode disaggregation Service level agreement

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A strict P99 TPOT SLA on bursty traffic is one of the hardest production constraints in LLM serving. It rules out most of the cost-optimal designs (high batch, on-demand capacity, mean-based scaling) because those choices buy throughput at the price of tail variance.

The right answer requires four architectural commitments: reserved capacity to eliminate infrastructure variance, prefill-decode disaggregation to eliminate cross-phase blocking, modest batch caps to keep per-stream TPOT predictable, and tail-based autoscaling to absorb bursts before the SLA breaks. Each lever targets a different source of tail variance.

This deep dive covers each lever in turn, explains why the other MCQ options break the SLA in specific ways, and closes with the operational realities of running disaggregated serving in production. By the end you should be able to design a latency-tight serving stack and defend each choice against the cost-optimal alternative.

Why P99 TPOT is the hardest serving SLA

TPOT (time per output token) is the steady-state per-token latency once decode is in progress. P99 TPOT is the value that 99% of decoded tokens fall under. A strict P99 SLA (say, 50 ms) forces every architectural choice toward variance reduction, often at the cost of throughput per dollar.

The SLA differs from TTFT (time to first token) in important ways. TTFT is dominated by prefill cost and queue wait; TPOT is dominated by decode step time and any interruptions to the steady decode rhythm. A serving stack can have great TTFT and terrible P99 TPOT if a long prefill blocks ongoing decodes every few minutes.

Bursty traffic compounds the difficulty. Steady-state TPOT can be tight if the batch size is stable and prefill traffic is uniform. Bursts introduce two stressors: spikes in queue depth (TTFT impact) and spikes in prefill arrival rate (TPOT impact on co-located stacks). The hardest SLAs combine strict P99 with bursty traffic, which is exactly the scenario in the question.

Reserved capacity: eliminating infrastructure variance

Prefill-decode disaggregation: the dominant lever

Batch caps and the bandwidth to compute crossover

Tail-based autoscaling and the burst absorption problem

Why A, C, and D each break the SLA

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

NVIDIA Dynamo (formerly Triton Inference Server v2) implements prefill-decode disaggregation as a first-class feature for latency-sensitive deployments.
Anthropic's Claude Opus 4.7 production serving disaggregates prefill onto dedicated H100 clusters and streams KV cache over NVLink to decode replicas.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does disaggregation handle the KV cache transfer between prefill and decode replicas?

Prefill emits the full KV cache for the prompt; decode needs that cache to start generating. The transfer is over NVLink (for intra-node) or InfiniBand (for inter-node). Cache size scales with prompt length and model size, often 100s of MB to GBs. Latency of transfer must be smaller than the TTFT budget. NVIDIA Dynamo and similar stacks use streaming transfers that overlap with the first decode steps.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Picking throughput-maximizing batch sizes for a latency-bound SLA. High batch trades P99 TPOT for tokens per dollar and breaks the SLA exactly when traffic spikes.

60 second bullets to scan on the way to the call.

Identify the four design pressures of a strict P99 TPOT SLA
Explain why prefill blocks decode in co-located serving