Describe a concrete, production runnable mechanism to detect hallucinations in a RAG answer (claims unsupported by the retrieved chunks), as the answer is generated or just after.

Short answer·Medium·4.0 · 0·~3 min·Asked atJump Trading Perplexity Uipath·Relevant atAnthropic

Attempt it

You are running a production RAG system. Describe a concrete mechanism (not 'just have humans review them') that flags when an answer contains claims unsupported by the retrieved chunks. Cover the inputs, the check, and what happens on a positive detection.

Free · 2 AI evals / day

Hallucination detection in RAG is the question: did the model say something that the retrieved chunks do not actually support? You cannot answer that by looking at the answer alone. You need both the answer and the chunks, and you need a check that compares them. The standard mechanism is **LLM-as-judge claim grounding**. After the generator produces an answer, split the answer into atomic claims (one fact per claim, sentence level is a good start). For each claim, call a smaller, cheaper LLM with the claim, the retrieved chunks, and a prompt like: 'Does this claim follow from these chunks? Answer yes, no, or partial. Quote the supporting span if yes.' Aggregate the per-claim verdicts into a faithfulness score, the fraction of claims supported. If the score is below a threshold (say 0.7), do one of three things: (a) auto-retry the generation with a stricter system prompt, (b) downgrade the answer to a 'sorry, I don't have enough info' fallback, or (c) log the case to a review queue. This costs one extra LLM call per answer. High-volume systems often sample 1-5% of traffic instead of running it on every answer.

Hallucination detection runs as a post-generation check that compares each claim in the answer against the retrieved chunks. The classic pattern is LLM-as-judge claim grounding. **The inputs are the answer and the chunks**. Checking the answer alone cannot detect hallucination. Hallucinations are mismatches between the answer and the source, so both are required. Some implementations also include the question to ground the relevance check. **The check decomposes the answer into atomic claims**, typically one per sentence. Either a small splitter LLM produces the list or a rule based splitter on sentence boundaries does the job. For each claim, a judge model is prompted with the claim, the chunks, and the question. The prompt asks whether the claim follows from these chunks. The judge answers yes, no, or partial, and quotes the supporting span on yes. The judge is a smaller LLM than the generator. Pair Claude Opus 4.7 with Claude Haiku 4.7, or pair GPT-5.5 with GPT-5.5-mini. **Aggregation produces a faithfulness score**, the fraction of claims marked supported. Below a threshold (commonly 0.7), the system takes one of three actions. It can auto-retry with a stricter grounding directive, downgrade to a refusal fallback ('I don't have enough info'), or log the example to a review queue for offline analysis. The cost is one extra LLM call per answer. High-volume systems sample 1-5 percent of traffic; high stakes systems run it on every answer. This is the same pattern that RAGAS, TruLens, and Patronus AI implement.

TL;DR

Split the answer into atomic claims, ask a smaller judge LLM whether each claim follows from the retrieved chunks, aggregate to a faithfulness score, and refuse, retry, or log when the score is below threshold.

Memory aid

Easy to grasp

Imagine a student wrote a five-sentence answer and you want to check whether each sentence is actually backed by the textbook pages you handed them. You go sentence by sentence and ask 'is this in the pages or not?'. If most sentences check out, you accept the answer. If a sentence is invented, you flag it. A production RAG system does exactly this with a smaller AI playing the role of the checker. The big AI writes the answer; the small AI compares each claim against the retrieved chunks. If the small AI flags too many unsupported claims, the system either refuses, retries with stricter instructions, or saves the case for a human reviewer.

Key concepts

RAG system design LLM evaluation Prompt engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Hallucination detection is the safety check that catches what retrieval and prompt grounding miss. A well designed RAG pipeline still produces answers that contain claims not supported by the retrieved chunks: subtle paraphrases that drift from the source, generator side pretraining knowledge that bleeds into the answer, confident assertions about facts the chunks did not actually contain. Without an explicit detector, these slip into production silently.

This deep dive walks through the canonical mechanism (LLM-as-judge claim grounding), what makes it work, the cost tradeoffs that shape deployment patterns, how it relates to offline evaluation frameworks like RAGAS and TruLens, and the senior nuances around judge selection, sampling strategy, and the gap between hallucination detection and answer relevance.

Why you cannot detect hallucinations from the answer alone

A common naive approach is to look at the answer's surface features: confidence language, hedging, length, perplexity, or self-reported uncertainty. None of these correlates reliably with hallucination.

Generator LLMs sound confident regardless of whether they are right. Hedging language ('I think', 'possibly') is a function of prompt style and temperature, not truth. Perplexity measures how surprising the output was to the model, which is uncorrelated with whether the output matches the source.

Hallucination is a relational property between two things: the answer and the source. The answer says X. The source either does or does not contain X. Detecting hallucination means comparing those two artifacts. Any check that looks at only one of them is solving a different problem.

The mechanism that works is a comparison check. Inputs: the generated answer and the retrieved chunks. Optionally also the question, to assess whether the chunks were relevant in the first place. Output: per-claim verdicts on whether each piece of the answer follows from the chunks. This shape is invariant across LLM-as-judge, NLI-based, and human evaluation approaches; only the verifier model changes.

Atomic claim decomposition

The judge: model choice and prompt design

Aggregation, threshold gating, and the three actions

Cost, sampling, and the offline online relationship

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import re

JUDGE_PROMPT = """You are a grounding judge.

Claim: {claim}

Retrieved chunks:
{chunks}

Does the claim follow from these chunks?
Answer one of: yes / no / partial.
If yes or partial, quote the supporting span.
If no, explain briefly."""

def split_claims(answer):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", answer) if s.strip()]

def judge_claim(claim, chunks, judge_llm):
    out = judge_llm.complete(JUDGE_PROMPT.format(claim=claim, chunks=chunks))
    return out.split()[0].lower()  # yes / no / partial

def faithfulness_score(answer, chunks, judge_llm="claude-haiku-4.7"):
    claims = split_claims(answer)
    verdicts = [judge_claim(c, chunks, judge_llm) for c in claims]
    supported = sum(1 for v in verdicts if v in ("yes", "partial"))
    return supported / max(1, len(claims))

def gated_answer(answer, chunks, threshold=0.7):
    score = faithfulness_score(answer, chunks)
    if score < threshold:
        return {"action": "refuse", "score": score,
                "answer": "I don't have enough info to answer that confidently."}
    return {"action": "accept", "score": score, "answer": answer}

Approach	Cost	Strength	Weakness
LLM-as-judge per claim	Extra LLM call per answer	Flexible, handles nuance	Latency and dollar cost
NLI model (DeBERTa, AlignScore)	Tens of ms per claim	Cheap, fast	Less nuanced on complex claims
Answer-level vibe check	One LLM call	Cheapest LLM-judge variant	Misses isolated hallucinations
No detection (trust generator)	Zero	No latency	Hallucinations reach users
Sampled detection (1-5% traffic)	Bounded extra cost	Catches trends, monitors drift	Misses per-user hallucinations
Full-coverage detection	Doubles per-answer cost	Catches every case	Expensive at scale

Real products, models, and research that use this idea.

RAGAS is the leading open source RAG evaluation framework; its faithfulness metric implements exactly this pattern (claim decomposition plus per-claim grounding check by a judge LLM).
TruLens by TruEra offers online and offline hallucination detection on RAG pipelines with hooks for popular LLM judges.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you detect that your judge model is itself unreliable?

Build a human labeled set of (answer, chunks, ground-truth verdict) tuples and measure judge accuracy quarterly. Compare against a second judge from a different model family for agreement. If agreement drops below a threshold, swap or retrain the judge.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Trying to detect hallucinations by looking at the answer alone or by measuring answer confidence. Hallucinations are answer versus source mismatches; both inputs are required.

60 second bullets to scan on the way to the call.

Why hallucination detection requires both the answer and the retrieved chunks as inputs
How per-claim decomposition catches isolated hallucinations an answer level check misses

Primary sources. Browse if you want the original framing.