P99 TTFT spikes every Tuesday at 9am, order the triage steps

**TTFT** (time to first token) is the latency from request arrival to the first generated token. P99 means the slowest 1% of requests. A weekly periodic spike, every Tuesday at 9am, points strongly at a calendar-driven cause, not a model bug. Triage cheapest first. The cheapest signal is queue depth from your existing dashboards. If the queue is deeper at 9am Tuesday than the same hour on other days, the system is starved of replicas relative to demand and TTFT spikes because requests are waiting before any GPU even sees them. This is the single most common explanation. The second-cheapest signal is the **input-length distribution**. Look at the P99 input length for Tuesday-morning requests vs baseline. Sometimes a weekly job, a Monday-night batch summarization, a Tuesday-morning report generator, sends pathologically long prompts (100K tokens). Long prompts dominate prefill time and push TTFT into the tail. Next check the autoscaler ramp: did new replicas come up *before* the surge or did they trail behind, leaving the system under-provisioned during the burst? Then look at noisy-neighbor signals on shared GPU nodes. Finally rule out upstream network issues, most expensive to investigate, least often the cause.

**TTFT triage** for a recurring time of week pattern follows a strict cheapest signal first ordering. Two checks resolve 80% of these incidents; the remaining three exist for the cases where the first two come up clean. First, queue depth at 9am Tuesday vs the same hour on other weekdays. A periodic spike with elevated queue depth is a capacity story: replicas are insufficient relative to instantaneous demand. The metric is sitting on your existing scheduler dashboard (vLLM, SGLang, TGI all expose it), no instrumentation needed. Second, the input-length distribution. Long prompts dominate prefill time, and prefill is the first-token cost. A weekly upstream job (Monday-night batch that overlaps with Tuesday traffic, a 9am scheduled report tool that suddenly fires 100K-token prompts) is a classic root cause that shifts the input-length distribution without changing request count. The check is one histogram comparison. Third, the autoscaler ramp. Even if queue depth and input lengths look normal, the spike can come from a lag in the autoscaler's response. HPA on a coarse metric with a slow polling period adds replicas after demand peaks, so the surge runs on yesterday's replica count. The fix is either to autoscale on a leading indicator (queue depth, batch-fullness) instead of a lagging one, or to pre-scale on a schedule for known traffic patterns. Fourth, noisy neighbors. Less common but real on shared GPU nodes, another workload's burst contends for SM cycles or KV-block memory. Fifth, upstream network, LB latency spikes, weekly TLS cert rotations, hardware-level events. These show up on infra dashboards independent of the model, and you check them last because they are the rarest cause of a recurring model-serving pattern.