Find the security flaw in how this multi-tenant RAG system enforces document permissions

Spot the error·Hard·4.0 · 0·~2 min·Asked atBain Midjourney Truera

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The flaw is that access control is enforced by a sentence in the prompt. The code searches the entire index for everyone, then tells the model: 'Ignore any chunk the user is not authorized to read.' By that point the forbidden chunks are already in the context window, and that is the whole problem. Once a forbidden chunk is in context, it can leak in several ways. The model may quote or summarize it despite the instruction. A prompt injection can coax the model past the 'ignore' rule. And the instruction is **unverifiable** — there is no way to prove the model withheld the data. Access control is a retrieval-layer concern, not a prompt concern. The decision about what a user may see belongs when you query the index, not after. The fix is to filter the index on the caller's permissions before ranking, so unauthorized documents are never fetched. Pass the user's access list as a filter on the search: only chunks the user is cleared for are candidates. If a forbidden chunk never reaches the model, there is nothing to leak and nothing to coax out. That is the difference between a real guarantee and a polite request.

The error is the access-control model: permissions are being enforced by a prompt instruction rather than at retrieval. The code runs an unfiltered search across the whole index for every user, then appends 'Ignore any chunk the user is not authorized to read' to the prompt. The forbidden chunks are already in the context window before the model is asked to disregard them, and that ordering is the bug. This fails for three independent reasons. First, the model may quote or paraphrase the forbidden chunk despite the instruction — generation is probabilistic, not obedient. Second, a prompt injection in any retrieved content can override the 'ignore' rule and surface the protected data. Third, the control is **unverifiable**: you cannot prove the model withheld the chunk, which fails any compliance review. The principle is that access control belongs at the retrieval layer, not the prompt layer. The fix is to filter the index on the caller's access list before ranking. Ideally this is a pre-filter, so the ANN search only ever considers authorized chunks. `chunks = index.search(embed(query), top_k=8, filter={"acl": user.allowed_groups})` Now a forbidden document is never fetched into context. There is nothing to quote, nothing to inject past, and nothing to audit-fail on. The leak path and the injection attack surface both disappear because the data is simply not present. A prompt instruction can be a defense in depth nicety on top, but the actual guarantee has to live where the documents are selected.

TL;DR

Access control is enforced by a prompt line telling the model to ignore unauthorized chunks. That's broken — chunks already in context can leak. Filter on the user's ACL at retrieval so they're never fetched.

Memory aid

Easy to grasp

Imagine a librarian who pulls every book in the building onto your desk, including ones you're not allowed to see, then tells you 'don't read the ones you shouldn't.' The secret is already sitting in front of you — you might glance at it, or someone could trick you into reading it aloud. The safe way is for the librarian to check your library card first and only bring the books you're cleared for. In this code, the system grabs documents for everyone and asks the AI to politely look away. Real access control checks who you are before fetching, so forbidden documents never reach the desk at all.

Key concepts

Retrieval-Augmented Generation Safety and Guardrails

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

This snippet looks reasonable at a glance, which is exactly why it is a good interview trap. It retrieves context, it acknowledges that some chunks are off-limits, and it tells the model to respect that. The instinct is to nod along. The senior instinct is to notice the ordering: the forbidden data is fetched into the prompt first, and only then is the model asked to ignore it. By that point the security decision has already been lost.

The flaw is not a bug in one line; it is a category error about where authorization lives. Access control is a deterministic property a system must enforce, and this code delegates it to a probabilistic text generator via a polite request. This deep dive establishes why a prompt instruction can never be an access-control mechanism, enumerates the concrete leak paths, explains the retrieval-layer fix and why pre-filtering specifically matters, and ends on the general rule that an LLM must never be the enforcement point for a security boundary.

Why a prompt instruction is not access control

Access control has a precise meaning in security: a deterministic, enforceable, auditable decision about whether a principal may access a resource. Authentication establishes who you are; authorization decides what you may see. Both are properties the system guarantees, not behaviors it requests.

The code in question does neither. It fetches every chunk that matches the query across the whole index — for every user, regardless of tenant or ownership — and then appends a sentence: 'Ignore any chunk the user is not authorized to read.' That sentence is a request to a language model, and a language model is the wrong component to hold a security boundary for one reason above all: its output is probabilistic. There is no mechanism that forces it to honor a negative instruction, and no privileged status that makes your instruction outrank text inside the retrieved chunks.

The deeper framing is the confused deputy. The retrieval layer acts with full privilege — it can read everything — and then hands the authorization decision down to a component that should never have been trusted with it. The fix is not to word the request better. It is to move the decision back to a layer that can enforce it deterministically, before the privileged fetch ever returns forbidden data.

The leak paths a forbidden chunk opens

The fix: enforce at retrieval with a pre-filter

The principle and the residual leak paths

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Multi-tenant enterprise RAG filters the vector index on the caller's groups so one tenant can never retrieve another's documents
OWASP LLM Top 10 covers sensitive-information disclosure that prompt-level filtering fails to prevent

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is a pre-filter on the ACL safer than retrieving top-k and then dropping unauthorized results?

A post-filter runs the ANN search over all vectors, including forbidden ones, then removes the unauthorized hits — which under-fills top-k (you asked for 8, got back 3 authorized) and still computes over data the user can't see, with timing or count side channels. A pre-filter restricts the candidate set before scoring, so unauthorized vectors are never considered and top-k is filled from authorized documents only.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Trusting a prompt line like 'ignore unauthorized chunks' to enforce permissions, when the forbidden data is already in the context window and can leak.

60 second bullets to scan on the way to the call.

Why a prompt instruction cannot enforce access control
The three ways a forbidden chunk in context can still leak

Primary sources. Browse if you want the original framing.