**TTFT** (time to first token) is the latency from request arrival to the first generated token. P99 means the slowest 1% of requests. A weekly periodic spike, every Tuesday at 9am, points strongly at a calendar-driven cause, not a model bug.
Triage cheapest first. The cheapest signal is queue depth from your existing dashboards. If the queue is deeper at 9am Tuesday than the same hour on other days, the system is starved of replicas relative to demand and TTFT spikes because requests are waiting before any GPU even sees them. This is the single most common explanation.
The second-cheapest signal is the **input-length distribution**. Look at the P99 input length for Tuesday-morning requests vs baseline. Sometimes a weekly job, a Monday-night batch summarization, a Tuesday-morning report generator, sends pathologically long prompts (100K tokens). Long prompts dominate prefill time and push TTFT into the tail.
Next check the autoscaler ramp: did new replicas come up *before* the surge or did they trail behind, leaving the system under-provisioned during the burst? Then look at noisy-neighbor signals on shared GPU nodes. Finally rule out upstream network issues, most expensive to investigate, least often the cause.
**TTFT triage** for a recurring time of week pattern follows a strict cheapest signal first ordering. Two checks resolve 80% of these incidents; the remaining three exist for the cases where the first two come up clean.
First, queue depth at 9am Tuesday vs the same hour on other weekdays. A periodic spike with elevated queue depth is a capacity story: replicas are insufficient relative to instantaneous demand. The metric is sitting on your existing scheduler dashboard (vLLM, SGLang, TGI all expose it), no instrumentation needed.
Second, the input-length distribution. Long prompts dominate prefill time, and prefill is the first-token cost. A weekly upstream job (Monday-night batch that overlaps with Tuesday traffic, a 9am scheduled report tool that suddenly fires 100K-token prompts) is a classic root cause that shifts the input-length distribution without changing request count. The check is one histogram comparison.
Third, the autoscaler ramp. Even if queue depth and input lengths look normal, the spike can come from a lag in the autoscaler's response. HPA on a coarse metric with a slow polling period adds replicas after demand peaks, so the surge runs on yesterday's replica count. The fix is either to autoscale on a leading indicator (queue depth, batch-fullness) instead of a lagging one, or to pre-scale on a schedule for known traffic patterns.
Fourth, noisy neighbors. Less common but real on shared GPU nodes, another workload's burst contends for SM cycles or KV-block memory.
Fifth, upstream network, LB latency spikes, weekly TLS cert rotations, hardware-level events. These show up on infra dashboards independent of the model, and you check them last because they are the rarest cause of a recurring model-serving pattern.
Quick reference
TL;DR
Triage cheapest signal first: queue depth and prompt-length distribution explain 80% of weekly-pattern TTFT spikes.
Picture a coffee shop that gets slammed every Tuesday at 9am. Before you tear apart the espresso machine, do the cheap checks. First, is the line at the door longer on Tuesdays? Second, is everyone suddenly ordering a 12-shot custom drink instead of a regular latte? Those two questions answer most cases. After that you can ask whether the barista was scheduled to arrive before or after the rush hit, whether someone else is using the same shared shelf for their stuff, or whether the front door's intercom is glitching. The order matters because some checks take a glance at a dashboard and others require pulling a barista off the line.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A one-time spike with no recurring pattern One-off events more often trace to deploys, single bad replicas, or upstream incidents, the time of week pattern triage tree does not apply.
Spikes correlated with a release window Start by checking deploy timing and rollback candidates; the recurring-pattern checklist is for calendar-driven events, not change-driven ones.
Decode-phase latency spikes (inter-token, not TTFT) Decode tail issues trace to KV-cache fragmentation, paged-attention block exhaustion, or batch-mix problems, a different triage tree.
Quick scan · 2–4 min
Everything important, quickly.
3 min: the cheapest signal first principle + queue depth as the first check + input-length distribution as the second + autoscaler ramp diagnosis + when to escalate to noisy-neighbor and network checks.
Real-world examples
Real products, models, and research that use this idea.
1vLLM and SGLang expose queue depth and waiting-request count on their /metrics endpoints, intended for autoscaler signals and exactly this kind of triage.
2Anthropic's, OpenAI's, and Together AI's status pages frequently note input length driven spikes around scheduled customer batch jobs.
3KEDA's external-metrics scalers in 2026 commonly autoscale LLM serving pods on queue depth or in-flight requests rather than CPU.
4Cursor and Anysphere have publicly discussed pre-scaling on-schedule for known business-hours traffic peaks rather than relying on reactive HPA.
5Datadog's LLM Observability product in 2026 surfaces P99 TTFT broken down by input-length bucket precisely because long prompts are the dominant tail driver.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you instrument to make this triage faster next time?
Sign in or go Pro to see the approach for this follow-up.
QIf queue depth is flat but TTFT still spikes on Tuesday morning, what does that rule in and out?
Sign in or go Pro to see the approach for this follow-up.
QWhy is autoscaling on queue depth strictly better than autoscaling on TTFT?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Jumping to the model itself or to GPU-level traces before checking queue depth and prompt-length distribution. Recurring time of week spikes are almost always a traffic-shape or autoscaler problem, not a model problem.