Your HNSW deployment has a recall@10 SLO of 0.95 and a p99 latency budget of 80 ms. Data volume grows weekly and the embedding distribution drifts as new document types arrive, so a hand tuned ef_search goes stale. Design an automatic ef_search tuning loop: what you measure, how you measure it cheaply, and how the control loop adjusts the knob safely.
Run a small Flat-ground-truth probe set against the live HNSW off the hot path, then close a bounded loop on ef_search with hysteresis and a hard latency clamp.
Imagine a kitchen with a chef who tastes the soup once an hour. The chef does not taste every bowl going out, that would be too slow. Instead the chef keeps a small set of trusted reference spoons and only tastes those, comparing them to a perfect reference batch made in advance. If the test bowls taste a little off, the chef stirs in a touch more salt and waits to taste again. If they taste too salty and the timer is running short, the chef holds steady. The chef never doubles the salt suddenly, never relies on one tasting, and never lets the soup miss the serving time. The recall probe is the tasting, the salt is ef_search, the timer is the latency budget, and the rule of small steps with waiting is hysteresis.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Auto-tuning ef_search looks deceptively simple. The knob is one integer, the SLO is one number, and the control law in a textbook would fit on a napkin. The reason production teams still get it wrong is that the loop is closing over a moving target. The corpus grows, the embedding distribution drifts, traffic mixes change, and the index's own internal structure ages with every batch of inserts and deletes. A naive controller built on a static ground-truth set will silently chase a stale signal and ship the SLO miss as a feature.
The interesting design work is not the controller. It is the lifecycle of the measurement signal and the guardrails around the knob. Get those two right and the controller can be three lines of code. Get either one wrong and a more clever controller just gets you to the wrong answer faster.
This walkthrough lays out the full loop end to end: the probe set and its refresh policy, the shadow-path measurement, the bracketed recall vs latency slope, the multiplicative controller with hysteresis, and the escalation paths when the knob runs out of range.
The recall signal: probe sets and ground-truth lifecycle
The only honest way to measure recall at scale is to compare ANN results against exact-search results for a fixed set of queries. The probe set is a few hundred representative queries selected to span the active query distribution, plus their true top-10 neighbors computed by Flat (brute-force) search over a snapshot of the current corpus.
Few hundred is a real number, not a hand wave. With 500 probe queries you can resolve recall@10 to about 0.005, which is fine for tracking a 0.95 SLO. Fewer probes and noise eats your signal; more and the cost of refreshing ground truth becomes its own problem.
The refresh policy is what people forget. Ground truth ages whenever the corpus changes. A 5% corpus turnover, a large bulk delete, a new document-type ingest, or a measurable shift in the query centroid all invalidate ground-truth top-10s. Build a small daemon that recomputes ground truth on any of those triggers. Otherwise the loop chases a fiction: the index is fine, the ground truth has drifted, and the controller raises ef_search until p99 explodes.
Multi-tenant deployments need per-tenant probe sets. A code-search tenant and a legal-search tenant share an index but live in different parts of embedding space; one set of probes will not generalize.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Pinecone serverless exposes a managed recall knob and uses internal probe-style telemetry to track recall drift across data plane reshards.
- Turbopuffer publishes recall vs latency curves derived from continuously refreshed ground-truth sets and rotates ef parameters per index automatically.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect that the probe set itself has gone stale before recall regressions show up?
Monitor probe-query embedding centroid drift vs live-query centroid, alert when KL or cosine drift exceeds a threshold, and schedule ground-truth recompute on the trigger.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Closing the loop on a stale ground-truth probe set so the tuner chases a fiction, or letting the controller raise ef_search past the latency SLO instead of escalating.
60 second bullets to scan on the way to the call.
Why a probe set with exact-search ground truth is the only honest recall signal at scale
How often to refresh ground truth and what events trigger an early refresh
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.