Describe a concrete, production runnable mechanism to detect hallucinations in a RAG answer (claims unsupported by the retrieved chunks), as the answer is generated or just after.
You are running a production RAG system. Describe a concrete mechanism (not 'just have humans review them') that flags when an answer contains claims unsupported by the retrieved chunks. Cover the inputs, the check, and what happens on a positive detection.
Free · 2 AI evals / day
Hallucination detection in RAG is the question: did the model say something that the retrieved chunks do not actually support? You cannot answer that by looking at the answer alone. You need both the answer and the chunks, and you need a check that compares them.
The standard mechanism is **LLM-as-judge claim grounding**. After the generator produces an answer, split the answer into atomic claims (one fact per claim, sentence level is a good start). For each claim, call a smaller, cheaper LLM with the claim, the retrieved chunks, and a prompt like: 'Does this claim follow from these chunks? Answer yes, no, or partial. Quote the supporting span if yes.'
Aggregate the per-claim verdicts into a faithfulness score, the fraction of claims supported. If the score is below a threshold (say 0.7), do one of three things: (a) auto-retry the generation with a stricter system prompt, (b) downgrade the answer to a 'sorry, I don't have enough info' fallback, or (c) log the case to a review queue.
This costs one extra LLM call per answer. High-volume systems often sample 1-5% of traffic instead of running it on every answer.
Hallucination detection runs as a post-generation check that compares each claim in the answer against the retrieved chunks. The classic pattern is LLM-as-judge claim grounding.
**The inputs are the answer and the chunks**. Checking the answer alone cannot detect hallucination. Hallucinations are mismatches between the answer and the source, so both are required. Some implementations also include the question to ground the relevance check.
**The check decomposes the answer into atomic claims**, typically one per sentence. Either a small splitter LLM produces the list or a rule based splitter on sentence boundaries does the job. For each claim, a judge model is prompted with the claim, the chunks, and the question. The prompt asks whether the claim follows from these chunks. The judge answers yes, no, or partial, and quotes the supporting span on yes. The judge is a smaller LLM than the generator. Pair Claude Opus 4.7 with Claude Haiku 4.7, or pair GPT-5.5 with GPT-5.5-mini.
**Aggregation produces a faithfulness score**, the fraction of claims marked supported. Below a threshold (commonly 0.7), the system takes one of three actions. It can auto-retry with a stricter grounding directive, downgrade to a refusal fallback ('I don't have enough info'), or log the example to a review queue for offline analysis.
The cost is one extra LLM call per answer. High-volume systems sample 1-5 percent of traffic; high stakes systems run it on every answer. This is the same pattern that RAGAS, TruLens, and Patronus AI implement.
Quick reference
TL;DR
Split the answer into atomic claims, ask a smaller judge LLM whether each claim follows from the retrieved chunks, aggregate to a faithfulness score, and refuse, retry, or log when the score is below threshold.
Imagine a student wrote a five-sentence answer and you want to check whether each sentence is actually backed by the textbook pages you handed them. You go sentence by sentence and ask 'is this in the pages or not?'. If most sentences check out, you accept the answer. If a sentence is invented, you flag it. A production RAG system does exactly this with a smaller AI playing the role of the checker. The big AI writes the answer; the small AI compares each claim against the retrieved chunks. If the small AI flags too many unsupported claims, the system either refuses, retries with stricter instructions, or saves the case for a human reviewer.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Strict sub-second total latency requirements on the hot path The extra LLM judge call adds 200-800ms; in voice agent or sub-second flows, defer detection to offline sampling instead of inline gating.
Single-source authoritative corpus where claims are trivially traceable If every chunk has structured metadata pointing to canonical entities, a rule based traceability check beats an LLM judge in both cost and reliability.
Creative or exploratory tasks where strict faithfulness is not the goal Brainstorming or open ended ideation outputs are not bounded by the chunks; faithfulness checks would gate every output and frustrate users.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name LLM-as-judge claim grounding, specify both inputs (answer plus chunks), describe per-claim decomposition and judge prompt, give the three gating actions, and close with the sampling versus full coverage cost tradeoff.
Code example
python
import re
JUDGE_PROMPT = """You are a grounding judge.
Claim: {claim}
Retrieved chunks:
{chunks}
Does the claim follow from these chunks?
Answer one of: yes / no / partial.
If yes or partial, quote the supporting span.
If no, explain briefly."""
def split_claims(answer):
return [s.strip() for s in re.split(r"(?<=[.!?])\s+", answer) if s.strip()]
def judge_claim(claim, chunks, judge_llm):
out = judge_llm.complete(JUDGE_PROMPT.format(claim=claim, chunks=chunks))
return out.split()[0].lower() # yes / no / partial
def faithfulness_score(answer, chunks, judge_llm="claude-haiku-4.7"):
claims = split_claims(answer)
verdicts = [judge_claim(c, chunks, judge_llm) for c in claims]
supported = sum(1 for v in verdicts if v in ("yes", "partial"))
return supported / max(1, len(claims))
def gated_answer(answer, chunks, threshold=0.7):
score = faithfulness_score(answer, chunks)
if score < threshold:
return {"action": "refuse", "score": score,
"answer": "I don't have enough info to answer that confidently."}
return {"action": "accept", "score": score, "answer": answer}
Side-by-side
Approach
Cost
Strength
Weakness
LLM-as-judge per claim
Extra LLM call per answer
Flexible, handles nuance
Latency and dollar cost
NLI model (DeBERTa, AlignScore)
Tens of ms per claim
Cheap, fast
Less nuanced on complex claims
Answer-level vibe check
One LLM call
Cheapest LLM-judge variant
Misses isolated hallucinations
No detection (trust generator)
Zero
No latency
Hallucinations reach users
Sampled detection (1-5% traffic)
Bounded extra cost
Catches trends, monitors drift
Misses per-user hallucinations
Full-coverage detection
Doubles per-answer cost
Catches every case
Expensive at scale
Real-world examples
Real products, models, and research that use this idea.
1RAGAS is the leading open source RAG evaluation framework; its faithfulness metric implements exactly this pattern (claim decomposition plus per-claim grounding check by a judge LLM).
2TruLens by TruEra offers online and offline hallucination detection on RAG pipelines with hooks for popular LLM judges.
3Patronus AI provides a hosted hallucination detection service (Lynx) that runs as a post-generation check on production RAG traffic.
4Anthropic Claude Haiku 4.7 and OpenAI GPT-5.5-mini are the 2026 references for small judge LLMs cheap enough to run on a meaningful fraction of production traffic.
5NLI models like microsoft/deberta-v3-large-mnli and AlignScore are the lightweight alternative to LLM judges; they run in tens of milliseconds and handle simple entailment well.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect that your judge model is itself unreliable?
Sign in or go Pro to see the approach for this follow-up.
QWalk me through how RAGAS faithfulness differs from a runtime hallucination detector.
Sign in or go Pro to see the approach for this follow-up.
QHow do NLI-based hallucination detectors compare to LLM-as-judge?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Trying to detect hallucinations by looking at the answer alone or by measuring answer confidence. Hallucinations are answer versus source mismatches; both inputs are required.