P99 TTFT spikes every Tuesday at 9am, order the triage steps

Order steps·Medium·4.0 · 0·~1 min·Relevant atOpenAI

Attempt it

1Compare queue depth at 9am Tuesday vs the same hour on other weekdays
2Rule out upstream issues: LB latency, TLS handshake spikes, weekly cert renewal jobs
3Check the autoscaler ramp curve: did replicas come up before or after the traffic surge crossed threshold?
4Look at the input-length distribution for Tuesday-morning requests vs baseline (any 100k-token prompts?)
5Inspect noisy-neighbor signals on shared GPU nodes (SM-util variance, KV-block contention)

**TTFT** (time to first token) is the latency from request arrival to the first generated token. P99 means the slowest 1% of requests. A weekly periodic spike, every Tuesday at 9am, points strongly at a calendar-driven cause, not a model bug. Triage cheapest first. The cheapest signal is queue depth from your existing dashboards. If the queue is deeper at 9am Tuesday than the same hour on other days, the system is starved of replicas relative to demand and TTFT spikes because requests are waiting before any GPU even sees them. This is the single most common explanation. The second-cheapest signal is the **input-length distribution**. Look at the P99 input length for Tuesday-morning requests vs baseline. Sometimes a weekly job, a Monday-night batch summarization, a Tuesday-morning report generator, sends pathologically long prompts (100K tokens). Long prompts dominate prefill time and push TTFT into the tail. Next check the autoscaler ramp: did new replicas come up *before* the surge or did they trail behind, leaving the system under-provisioned during the burst? Then look at noisy-neighbor signals on shared GPU nodes. Finally rule out upstream network issues, most expensive to investigate, least often the cause.

**TTFT triage** for a recurring time of week pattern follows a strict cheapest signal first ordering. Two checks resolve 80% of these incidents; the remaining three exist for the cases where the first two come up clean. First, queue depth at 9am Tuesday vs the same hour on other weekdays. A periodic spike with elevated queue depth is a capacity story: replicas are insufficient relative to instantaneous demand. The metric is sitting on your existing scheduler dashboard (vLLM, SGLang, TGI all expose it), no instrumentation needed. Second, the input-length distribution. Long prompts dominate prefill time, and prefill is the first-token cost. A weekly upstream job (Monday-night batch that overlaps with Tuesday traffic, a 9am scheduled report tool that suddenly fires 100K-token prompts) is a classic root cause that shifts the input-length distribution without changing request count. The check is one histogram comparison. Third, the autoscaler ramp. Even if queue depth and input lengths look normal, the spike can come from a lag in the autoscaler's response. HPA on a coarse metric with a slow polling period adds replicas after demand peaks, so the surge runs on yesterday's replica count. The fix is either to autoscale on a leading indicator (queue depth, batch-fullness) instead of a lagging one, or to pre-scale on a schedule for known traffic patterns. Fourth, noisy neighbors. Less common but real on shared GPU nodes, another workload's burst contends for SM cycles or KV-block memory. Fifth, upstream network, LB latency spikes, weekly TLS cert rotations, hardware-level events. These show up on infra dashboards independent of the model, and you check them last because they are the rarest cause of a recurring model-serving pattern.

Quick reference

TL;DR

Triage cheapest signal first: queue depth and prompt-length distribution explain 80% of weekly-pattern TTFT spikes.

Memory aid

Easy to grasp

Picture a coffee shop that gets slammed every Tuesday at 9am. Before you tear apart the espresso machine, do the cheap checks. First, is the line at the door longer on Tuesdays? Second, is everyone suddenly ordering a 12-shot custom drink instead of a regular latte? Those two questions answer most cases. After that you can ask whether the barista was scheduled to arrive before or after the rush hit, whether someone else is using the same shared shelf for their stuff, or whether the front door's intercom is glitching. The order matters because some checks take a glance at a dashboard and others require pulling a barista off the line.

Key concepts

Time to first token Production debugging Autoscaling Observability

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: the cheapest signal first principle + queue depth as the first check + input-length distribution as the second + autoscaler ramp diagnosis + when to escalate to noisy-neighbor and network checks.

Real-world examples

Real products, models, and research that use this idea.

vLLM and SGLang expose queue depth and waiting-request count on their /metrics endpoints, intended for autoscaler signals and exactly this kind of triage.
Anthropic's, OpenAI's, and Together AI's status pages frequently note input length driven spikes around scheduled customer batch jobs.
KEDA's external-metrics scalers in 2026 commonly autoscale LLM serving pods on queue depth or in-flight requests rather than CPU.
Cursor and Anysphere have publicly discussed pre-scaling on-schedule for known business-hours traffic peaks rather than relying on reactive HPA.
Datadog's LLM Observability product in 2026 surfaces P99 TTFT broken down by input-length bucket precisely because long prompts are the dominant tail driver.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you instrument to make this triage faster next time?

QIf queue depth is flat but TTFT still spikes on Tuesday morning, what does that rule in and out?

QWhy is autoscaling on queue depth strictly better than autoscaling on TTFT?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Jumping to the model itself or to GPU-level traces before checking queue depth and prompt-length distribution. Recurring time of week spikes are almost always a traffic-shape or autoscaler problem, not a model problem.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.