Describe a concrete, production runnable mechanism to detect hallucinations in a RAG answer (claims unsupported by the retrieved chunks), as the answer is generated or just after.

Short answer·Medium·4.0 · 0·~3 min·Asked atPerplexity·Relevant atAnthropic

Attempt it

You are running a production RAG system. Describe a concrete mechanism (not 'just have humans review them') that flags when an answer contains claims unsupported by the retrieved chunks. Cover the inputs, the check, and what happens on a positive detection.

Free · 2 AI evals / day

Hallucination detection in RAG is the question: did the model say something that the retrieved chunks do not actually support? You cannot answer that by looking at the answer alone. You need both the answer and the chunks, and you need a check that compares them. The standard mechanism is **LLM-as-judge claim grounding**. After the generator produces an answer, split the answer into atomic claims (one fact per claim, sentence level is a good start). For each claim, call a smaller, cheaper LLM with the claim, the retrieved chunks, and a prompt like: 'Does this claim follow from these chunks? Answer yes, no, or partial. Quote the supporting span if yes.' Aggregate the per-claim verdicts into a faithfulness score, the fraction of claims supported. If the score is below a threshold (say 0.7), do one of three things: (a) auto-retry the generation with a stricter system prompt, (b) downgrade the answer to a 'sorry, I don't have enough info' fallback, or (c) log the case to a review queue. This costs one extra LLM call per answer. High-volume systems often sample 1-5% of traffic instead of running it on every answer.

Hallucination detection runs as a post-generation check that compares each claim in the answer against the retrieved chunks. The classic pattern is LLM-as-judge claim grounding. **The inputs are the answer and the chunks**. Checking the answer alone cannot detect hallucination. Hallucinations are mismatches between the answer and the source, so both are required. Some implementations also include the question to ground the relevance check. **The check decomposes the answer into atomic claims**, typically one per sentence. Either a small splitter LLM produces the list or a rule based splitter on sentence boundaries does the job. For each claim, a judge model is prompted with the claim, the chunks, and the question. The prompt asks whether the claim follows from these chunks. The judge answers yes, no, or partial, and quotes the supporting span on yes. The judge is a smaller LLM than the generator. Pair Claude Opus 4.7 with Claude Haiku 4.7, or pair GPT-5.5 with GPT-5.5-mini. **Aggregation produces a faithfulness score**, the fraction of claims marked supported. Below a threshold (commonly 0.7), the system takes one of three actions. It can auto-retry with a stricter grounding directive, downgrade to a refusal fallback ('I don't have enough info'), or log the example to a review queue for offline analysis. The cost is one extra LLM call per answer. High-volume systems sample 1-5 percent of traffic; high stakes systems run it on every answer. This is the same pattern that RAGAS, TruLens, and Patronus AI implement.

Quick reference

TL;DR

Split the answer into atomic claims, ask a smaller judge LLM whether each claim follows from the retrieved chunks, aggregate to a faithfulness score, and refuse, retry, or log when the score is below threshold.

Memory aid

Easy to grasp

Imagine a student wrote a five-sentence answer and you want to check whether each sentence is actually backed by the textbook pages you handed them. You go sentence by sentence and ask 'is this in the pages or not?'. If most sentences check out, you accept the answer. If a sentence is invented, you flag it. A production RAG system does exactly this with a smaller AI playing the role of the checker. The big AI writes the answer; the small AI compares each claim against the retrieved chunks. If the small AI flags too many unsupported claims, the system either refuses, retries with stricter instructions, or saves the case for a human reviewer.

Key concepts

RAG system design LLM evaluation Prompt engineering

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: name LLM-as-judge claim grounding, specify both inputs (answer plus chunks), describe per-claim decomposition and judge prompt, give the three gating actions, and close with the sampling versus full coverage cost tradeoff.

Code example

python

import re

JUDGE_PROMPT = """You are a grounding judge.

Claim: {claim}

Retrieved chunks:
{chunks}

Does the claim follow from these chunks?
Answer one of: yes / no / partial.
If yes or partial, quote the supporting span.
If no, explain briefly."""

def split_claims(answer):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", answer) if s.strip()]

def judge_claim(claim, chunks, judge_llm):
    out = judge_llm.complete(JUDGE_PROMPT.format(claim=claim, chunks=chunks))
    return out.split()[0].lower()  # yes / no / partial

def faithfulness_score(answer, chunks, judge_llm="claude-haiku-4.7"):
    claims = split_claims(answer)
    verdicts = [judge_claim(c, chunks, judge_llm) for c in claims]
    supported = sum(1 for v in verdicts if v in ("yes", "partial"))
    return supported / max(1, len(claims))

def gated_answer(answer, chunks, threshold=0.7):
    score = faithfulness_score(answer, chunks)
    if score < threshold:
        return {"action": "refuse", "score": score,
                "answer": "I don't have enough info to answer that confidently."}
    return {"action": "accept", "score": score, "answer": answer}

Side-by-side

Approach	Cost	Strength	Weakness
LLM-as-judge per claim	Extra LLM call per answer	Flexible, handles nuance	Latency and dollar cost
NLI model (DeBERTa, AlignScore)	Tens of ms per claim	Cheap, fast	Less nuanced on complex claims
Answer-level vibe check	One LLM call	Cheapest LLM-judge variant	Misses isolated hallucinations
No detection (trust generator)	Zero	No latency	Hallucinations reach users
Sampled detection (1-5% traffic)	Bounded extra cost	Catches trends, monitors drift	Misses per-user hallucinations
Full-coverage detection	Doubles per-answer cost	Catches every case	Expensive at scale

Real-world examples

Real products, models, and research that use this idea.

RAGAS is the leading open source RAG evaluation framework; its faithfulness metric implements exactly this pattern (claim decomposition plus per-claim grounding check by a judge LLM).
TruLens by TruEra offers online and offline hallucination detection on RAG pipelines with hooks for popular LLM judges.
Patronus AI provides a hosted hallucination detection service (Lynx) that runs as a post-generation check on production RAG traffic.
Anthropic Claude Haiku 4.7 and OpenAI GPT-5.5-mini are the 2026 references for small judge LLMs cheap enough to run on a meaningful fraction of production traffic.
NLI models like microsoft/deberta-v3-large-mnli and AlignScore are the lightweight alternative to LLM judges; they run in tens of milliseconds and handle simple entailment well.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you detect that your judge model is itself unreliable?

QWalk me through how RAGAS faithfulness differs from a runtime hallucination detector.

QHow do NLI-based hallucination detectors compare to LLM-as-judge?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Trying to detect hallucinations by looking at the answer alone or by measuring answer confidence. Hallucinations are answer versus source mismatches; both inputs are required.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.