Two retrieved chunks contradict each other on the same fact. What should the system do?

Short answer·Hard·4.0 · 0·~3 min·Asked atElevenlabs Ey Weaviate

Attempt it

Retrieval surfaces chunks that directly contradict each other, for example an outdated policy and the current one, or two specs that disagree. Explain how a production RAG system should handle the conflict and why silently picking one answer is the wrong default.

Free · 2 AI evals / day

Retrieval often pulls back chunks that disagree. One chunk quotes the old policy, another quotes the current one. Two product specs list different limits. The dangerous default is for the model to quietly pick one and present it as fact, because the user has no way to know a second source said the opposite. The right behavior depends on whether the system has a signal to break the tie. The first good move is to resolve by **recency**: if the conflict is a versioning issue, prefer the document with the newer effective date or timestamp. The second is to resolve by **authority**: if some sources outrank others, prefer the official spec over a forum post. When neither signal cleanly decides, do not guess. Surface the conflict: say the sources disagree, show each answer, and cite both so the user can judge. That turns a silent failure into an honest one. All of this depends on groundwork at ingestion. The chunks must carry recency and authority as metadata, and citations must be tracked through to generation. Without that, the system has nothing to reason with and falls back to a coin flip dressed up as confidence.

A production RAG system must treat contradicting chunks as a first-class case, not let the language model resolve them by accident. The failure mode to avoid is **silent collapse**: the model blends or favors one chunk and emits a single confident answer. Whichever side it picked may be wrong, and the user cannot tell, so the system has manufactured a confidently wrong response. There are two principled resolution strategies. Resolve by recency when the conflict is a versioning issue — prefer the document with the newer effective date or last-updated timestamp, since the old policy is simply stale. Resolve by authority when sources have a clear ranking — an official spec outranks a community post, a signed contract outranks a draft. These rules only fire when the metadata supports them. When no signal cleanly decides, the correct move is to surface the disagreement. State that sources conflict, present each position, and cite both so the user can adjudicate. Abstaining from a false certainty is better than guessing. None of this works without upstream investment. Ingestion has to capture recency and authority as structured metadata on every chunk, and citation tracking has to thread source identity from retrieval through to the generated answer. The system can only reason about a conflict it can see, and it can only let the user judge a source it can name.

TL;DR

When retrieved chunks contradict, resolve by recency or authority metadata where a signal cleanly decides, and otherwise surface the conflict with citations — never silently pick one side as confident fact.

Memory aid

Easy to grasp

Imagine you ask two coworkers what the refund window is, and one says 30 days while the other says 14. A bad assistant just blurts out one number and sounds sure. A good assistant checks who is more trustworthy — maybe one is quoting last year's handbook and the other the current one — and if it still cannot tell, it says, 'These two notes disagree: one says 30 days, the other 14, here is where each came from.' That is what a careful RAG system does: it uses dates and source rank to break the tie, and when nothing breaks it, it shows you both and lets you decide instead of guessing for you.

Key concepts

Retrieval-Augmented Generation LLM System Design

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Most RAG failure analysis focuses on retrieval recall — did we fetch the right chunk? Conflicting chunks are a subtler and more dangerous case: retrieval did its job and surfaced two relevant chunks, but they disagree. Now the burden shifts to the generation layer, and the default behavior of a language model is exactly wrong. It will smooth the contradiction into one fluent, confident sentence, because that is what fluent text generation does.

This question matters because the cost of getting it wrong is asymmetric and corrosive. A user who later discovers that a cited source said the opposite of what the assistant told them does not just distrust that one answer — they distrust the whole system. This deep dive walks through why silent collapse happens, the two deterministic resolution strategies and their limits, the honest fallback of surfacing disagreement, and the ingestion-side groundwork that makes any of it possible.

Why the model collapses contradictions by default

A retriever scores chunks by semantic similarity to the query. Two chunks that both discuss the refund window will both score high, regardless of whether they agree. Cosine similarity has no notion of truth or recency; it measures topical relevance. So the contradicting pair lands in the context window together, side by side.

The generation step then does what decoders do: it produces the most probable continuation. If one chunk says 30 days and the other says 14, the model will usually latch onto whichever phrasing dominates the local context or simply pick one and write a confident sentence around it. It rarely flags the disagreement on its own, because nothing in the objective rewards hedging.

The key realization is that the conflict is invisible to the parts of the pipeline that could fix it cheaply. The retriever does not know the chunks disagree; it only knows they are relevant. The decoder treats both as equally trustworthy context. Unless you add an explicit step that compares claims and consults metadata, the contradiction resolves itself by accident — and an accidental resolution is indistinguishable, to the user, from a deliberate confident lie.

There is a second-order trap worth naming. Increasing top-k makes this worse, not better. Pull back 3 chunks and you might get one version of the fact; pull back 10 and you raise the odds of surfacing both the current and the superseded version together. So the very move teams reach for to improve recall — retrieve more — increases the rate of silent contradictions reaching the model. That is the opposite of what intuition predicts, and it is why conflict handling has to be a designed step rather than something you hope better retrieval will avoid.

Resolving by recency: the versioning case

Resolving by authority: the credibility case

Surfacing the conflict and the ingestion groundwork

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Perplexity surfaces multiple cited sources rather than collapsing conflicting search results into one unattributed claim, letting the reader weigh disagreements
Enterprise policy assistants prefer the document with the newest effective date when an old HR policy and its revision both get retrieved

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you automatically detect that two retrieved chunks contradict each other before generation?

Use a claim-extraction plus entailment pass: extract the key assertions from the top chunks and check pairwise whether any assert incompatible values for the same field or entity. A natural language inference model or an LLM judge can flag contradiction versus agreement. Run it on the small top-k set so the cost stays bounded, and only trigger the resolution tree when a contradiction is detected.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Letting the model silently favor whichever contradicting chunk landed first in the context, producing a confidently wrong answer that the user only catches when they read the other source.

60 second bullets to scan on the way to the call.

Why silently collapsing a contradiction is worse than admitting one
How recency metadata resolves a versioning conflict

Primary sources. Browse if you want the original framing.