Design document Q&A over 10 million docs where every user sees a different subset
Design a document Q&A system over 10 million enterprise documents with per document ACLs that change hourly. Retrieval must feel instant, and a user must never see content, citations, or even hints of documents they cannot open. Walk through index design, permission enforcement, freshness, and what you refuse to cache.
Push ACLs into the vector search as a pre-filter, shard by tenant, sync permissions via change events, and never cache responses across users.
Imagine a giant library where each book has a list of who can read it. The wrong way is to grab the ten best books matching a question and then check whether the reader is allowed to open them. By that time, the librarian has already mentioned the title of a secret book the reader is not supposed to know exists. The right way is to check the access list before searching at all. The librarian only looks among the books this specific reader can open, finds the ten best from that smaller pile, and the secret books never come up in conversation. The reader gets a fast, accurate answer, and the system never accidentally hints that other books are out there.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question is a trap dressed as a system design. A candidate who reaches for clever ranking, hybrid retrieval, and reranker stacks before saying the word permissions has already failed it. The hard requirement is that a user must never see content, citations, or even hints of documents they cannot open. That hard requirement collapses the design space.
The core insight is that the permission check has exactly one safe location: inside the similarity search, as a pre-filter on the ANN index. Every other location either leaks or destroys recall. Once that is fixed, the rest of the design (sharding, freshness, caching, observability) follows from standard distributed-retrieval engineering, with one twist. Every cache and every shard plan must respect the permission boundary.
This deep dive walks the four layers in turn: enforcement placement, sharding for 10M-plus chunks, ACL freshness against hourly churn, and what gets cached versus what does not.
Why ACLs must run inside the search
Three candidate placements exist and only one is defensible. Prompt-level enforcement asks the model to leave out restricted content. That is not security; it is etiquette. A single prompt injection or a small model error reveals everything. Post-retrieval filtering takes the global top-k and drops chunks the user cannot read. This works only when nothing restricted ever outranks authorized content, which is exactly the case real adversaries probe.
The failure modes of post-filtering are quantifiable. Recall: when a restricted chunk wins top-1, the user effectively loses a slot. With ten top-k slots and twenty percent of the corpus restricted, a representative query returns eight authorized chunks instead of ten. Side channel: the result count fluctuates with the restricted-content density of the query topic. A user who routinely sees ten citations and suddenly sees five has just learned that five restricted chunks are semantically near their query. Repeated queries narrow the leaked set.
Pre-filtered retrieval avoids both. The ANN index applies the filter expression before scoring, returning ten authorized chunks every time. Recall is preserved; the result count carries no signal about restricted content.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Glean and Atlassian Rovo enforce per-document ACLs by indexing identity metadata directly with their vector embeddings
- AWS Bedrock Knowledge Bases supports metadata filters on vector queries so permission predicates run inside the search
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you verify ACL enforcement actually works in production?
Stand up canary documents with known restricted ACLs, query them as unauthorized synthetic users on every deploy, and alert if any chunk, citation, or hint surfaces. Pair with per-release red-team evals that fish for known restricted content.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Filtering authorized chunks after retrieval feels simple but leaks the existence of restricted documents through result-count variation and wrecks recall when popular restricted documents crowd the top-k.
60 second bullets to scan on the way to the call.
Why pre-filtered retrieval beats post-filter on both recall and side-channel grounds
How group-based ACL metadata keeps filter cardinality manageable at scale
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.