During a background reindex, p99 latency triples while p50 barely moves. Pick the explanation.

MCQ·Medium·4.0 · 0·~1 min·Asked atArize Ai Datadog Workday

Attempt it

When p50 stays healthy but p99 triples, the shape of the problem is contention, not algorithmic slowdown. **Why the median is fine.** Most queries during the rebuild still find a free CPU core, hit cached vectors, and finish in their normal time. The rebuild is running on some threads, but most query work runs on others, so the typical query path is unaffected. **Why the tail explodes.** A fraction of queries arrive at a bad moment. They collide with a build thread for CPU. They miss the page cache because the rebuild just evicted the pages they needed. They queue behind a burst of segment-merge I/O. Each of these adds tens to hundreds of milliseconds of stall to the few unlucky queries, and the tail percentile is exactly what measures unlucky queries. The other options are wrong for clear reasons. Search does not fall back to brute force during a rebuild; the existing index keeps serving while the new one is being built. Recall and latency are independent axes; a less recall-accurate index can actually be faster, and a more accurate one slower. And dismissing tail metrics during maintenance windows is how SLO breaches reach users.

**Resource contention** is the right diagnosis because the latency shape (p50 healthy, p99 tripled) is the textbook signature. A background reindex consumes: - **CPU.** Build threads compete with query threads for cores. The OS scheduler usually gives queries enough time, but the slowest fraction get preempted and queued. - **Memory bandwidth.** The rebuild is reading every vector and writing the new index. Memory bandwidth is finite per socket; queries running concurrently get fewer effective bytes per second on the slowest paths. - **Page cache.** Old segments evicted, new segments brought in. Queries that wanted pages just evicted suffer cold reads from disk, adding milliseconds to tens of milliseconds per query. - **Disk I/O.** Sustained sequential writes from the rebuild compete with random reads from queries. The disk queue length spikes during compaction bursts; queries in the wrong place at the wrong time wait the queue out. None of these affect every query, which is why p50 is fine. All of them affect the unlucky few, which is why p99 triples. The operational fix is to throttle the rebuild (lower ingest threads, cap I/O, schedule during off-peak), to keep the rebuild on a separate node and switch via alias (Qdrant aliases, Pinecone namespaces) once it is built, or to accept the tail spike and time it for a maintenance window when SLO budgets allow it. Pretending tail latency does not matter during rebuilds is how SLO budgets get burned silently.

TL;DR

Healthy p50 with a tripled p99 is the signature of resource contention: most queries miss the rebuild, but the unlucky few collide with build threads or cache eviction and stall hard.

Memory aid

Easy to grasp

Imagine a kitchen where the staff usually plates orders quickly because each cook has a clear counter. Now the manager starts a deep clean of one section in the background, taking over an oven and one of the prep stations. Most orders flow through the unused stations just fine and arrive on time. But a few orders need the section being cleaned. Those orders wait for the deep clean to pause, or get bumped from station to station while space frees up, so they take much longer than usual. The median order looks healthy. The slowest few are terrible. That is what tail latency measures, and that is what an index rebuild does to queries.

Key concepts

Vector Databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The latency-shape mismatch (median fine, tail terrible) is one of the most diagnostic signals in distributed systems. It points at contention rather than at algorithmic change, and it shows up wherever a background workload shares finite resources with a foreground workload.

Vector index rebuilds are a particularly clean example because the rebuild is bandwidth-heavy, cache-hostile, and CPU-bound all at once. This section walks through why the latency distribution behaves that way, how vector engines specifically suffer the contention, and what operational patterns make the rebuild invisible to live traffic.

Why latency is right-skewed at steady state

A query against a vector index does the same algorithmic work every time, but the wall-clock time varies dramatically. The variance comes from the system around the algorithm.

Cache state. A query that hits a hot page in memory finishes in microseconds. A query that misses and reads from disk finishes in milliseconds. The ratio is roughly 1000x.

Scheduler luck. A query that runs on a core with no contention finishes uninterrupted. A query that gets preempted mid-flight by another process waits its quantum out.

NUMA effects. On a multi-socket box, accessing memory across sockets is 1.5 to 3x slower than local access. The query that lands on the wrong core has higher per-access cost.

Garbage collection or memory allocator pauses. Brief stalls of single-digit milliseconds for application-level work hit some queries and not others.

At steady state, these slow paths are rare, so p50 sits near the algorithmic floor. The right tail is where the slow paths accumulate; p99 might be 5 to 20x p50 in a healthy production system.

When you add a background workload, the slow paths get slower. The cache misses get longer because the rebuild evicted more pages. The scheduler preemptions get worse because more threads are competing. The tail grows proportionally to the worst-case stall the new contention adds, while the median barely moves.

What an HNSW rebuild actually does to the box

Why the other options are wrong

Operational patterns that fix the tail

Why reindex spikes p99 and the 2026 mitigations

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Qdrant supports aliases so a rebuilt collection on a sidecar node can be promoted atomically, avoiding in-place rebuild on the serving node
pgvector HNSW build is single-threaded by default and saturates one CPU and the page cache; teams run it on a maintenance replica before promoting

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you architect rebuilds so live serving nodes never see the load?

Build on a separate replica or a fresh node, then promote via alias or read-path switch. Qdrant aliases, Pinecone namespaces, and Elasticsearch index aliases all support this. The promotion is atomic and the live nodes never burned CPU on the build.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming p50 and p99 move together, then dismissing the rebuild tail spike as noise instead of treating it as the contention signal it actually is.

60 second bullets to scan on the way to the call.

Why latency distributions are right-skewed even at steady state
Which resources an HNSW rebuild competes for (CPU, memory bandwidth, page cache, disk I/O)

Primary sources. Browse if you want the original framing.