Select the signals a production RAG system must log to make failures debuggable

Multi-select·Medium·4.0 · 0·~1 min·Asked atIntuit Pinecone Sourcegraph

Attempt it

RAG fails in stages, so good logging has to capture each stage well enough that you can replay a request and point to the one that broke. Four signal groups do that. First, log the **incoming query and any rewritten or expanded form** you actually sent to the retriever. If query rewriting mangled the question, this is where you see it. Second, log the retrieved chunk IDs and their similarity or rerank scores. This tells you whether the right document was even found, and if it was, whether ranking pushed it down or kept it on top. Third, log the fully assembled prompt and the model's generated answer. If the right chunk was retrieved but the answer ignored it, the prompt and answer side by side reveal that. Fourth, log per-stage latency and token cost, plus any user feedback like a thumbs-up or thumbs-down. These tie quality and performance together. The two wrong choices: logging only the final answer leaves you blind to which stage failed, and dumping every chunk's raw embedding vector on every request is huge, static data that adds nothing a chunk ID does not already give you.

The goal of RAG observability is *failure localization*. When an answer is wrong, you need to replay the request and decide whether retrieval, ranking, or generation broke. That dictates what to log. You keep four signals. The first is the incoming query and the rewritten form sent to the retriever. Query transformation is a real failure source you cannot debug from the original alone. Second, the retrieved chunk IDs with similarity and rerank scores, which separate a retrieval miss (chunk never found) from a ranking miss (found but buried below the cutoff). Third, the assembled prompt and the generated answer, which catch the case where the right context was present but the model ignored it. Fourth, per-stage latency, token cost, and any user feedback signal, which connect quality to performance and give you a label for offline eval. With those four, a wrong answer becomes a guided investigation: was the needed chunk retrieved? If no, fix the retriever. If yes but low-ranked, fix ranking. If yes and well-ranked but the answer is wrong, fix the prompt or model. There are two traps. Logging only the final answer is the classic anti-pattern — the record that explains the least, because the failure already happened upstream. And dumping the raw embedding vector of every chunk per request is enormous and redundant: the vectors are static and live in the index. It adds no signal a chunk ID does not, since the ID lets you look the vector up if needed.

TL;DR

Log the query and its rewrite, retrieved chunk IDs with scores, the prompt and answer, and per-stage latency, cost, and feedback — enough to replay a request and localize the fault.

Memory aid

Easy to grasp

Imagine a relay race where the baton gets dropped but you only filmed the finish line. You see the team lost, but you have no idea which runner fumbled. To actually fix it, you need a camera at each handoff. RAG logging works the same way. The answer is the finish line. To debug a bad answer, you need a record at each handoff: what the question became, which documents were grabbed and how well they scored, what got handed to the model, and how long and how much each leg cost. With those, you can rewind the race and point to the exact runner who dropped the baton.

Key concepts

RAG LLM System Design LLM Evaluation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A RAG answer is the end of an assembly line: rewrite the query, embed it, retrieve candidates, rerank them, build a prompt, generate. When the answer is wrong, the failure already happened somewhere up that line. The entire job of observability is to let you walk back along it and find the station that broke.

That reframes the multi-select. The question is not really 'what data exists?' — it is 'what data lets you replay a request and localize its failure to one stage?' Some fields are diagnostic gold; some are the symptom dressed up as a log; and some are bulk data that belongs in the index, not the request trace.

This deep dive works through why the four winning signals each isolate a different stage, why the two distractors fail on different grounds, and how production teams in 2026 wire this up with tracing so the same data both debugs incidents and feeds offline evaluation. The throughline: log for the question you will be forced to answer under pressure — which stage do I fix?

Failure localization is the design goal

RAG has at least three stages that can each independently ruin an answer. Retrieval can fail to surface the chunk that holds the answer. Ranking can surface it but bury it below the top-k cutoff so it never reaches the prompt. Generation can receive the right chunk and still hallucinate, get lost in the middle of a long context, or follow an injected instruction. Three stages, three distinct fixes — different team, different code path, different test.

If your telemetry cannot tell these apart, every incident degrades into guessing. You see a wrong answer, you have a hunch, you tweak the chunk size, and you hope. That is not debugging; it is superstition.

So the design rule is simple: log enough per request to replay it and attribute the failure. Replay means you could reconstruct what each stage saw and produced. Attribute means the logged fields directly answer 'was the right chunk retrieved?', 'did it survive ranking?', and 'did the model use it?'. Every field worth keeping earns its place by advancing one of those three questions. Every field that does not is either noise or cost.

Walking the four winning signals

Why the two distractors are wrong on different grounds

Wiring it up: tracing in the 2026 stack

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Signal	Log it?	What it lets you diagnose
Query + rewritten form	Yes	Query transformation mangling recall
Chunk IDs + scores	Yes	Retrieval miss vs ranking miss
Assembled prompt + answer	Yes	Context present but model ignored it
Per-stage latency/cost/feedback	Yes	Performance and quality, with labels
Final answer only	No	Symptom only — cannot localize cause
Raw vector of every chunk/request	No	Nothing a chunk ID does not already give

Real products, models, and research that use this idea.

LangSmith traces each RAG step as a nested run, capturing the rewritten query, retrieved documents with scores, the final prompt, the answer, and per-step latency and token cost.
Langfuse models a request as a trace of spans (retriever, reranker, generation) and attaches user feedback scores, then feeds those traces into dataset-based evaluation.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you use logged scores to separate a retrieval miss from a ranking miss?

Build a small labeled set where you know which chunk holds the answer. For each query check whether that chunk id appears anywhere in the retrieved set: absent means a retrieval miss (fix embeddings, chunking, or index recall); present but below the top-k cutoff means a ranking miss (fix the reranker or widen the funnel). The score column tells you which without re-running the query.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Logging only the final answer to save space. When the answer is wrong you cannot tell whether retrieval missed the chunk, ranking buried it, or the model ignored it — the one record you kept is the one that explains the least.

60 second bullets to scan on the way to the call.

State the goal of RAG observability in one phrase: replay and localize the failing stage.
List the four signal groups worth logging on every request.

Primary sources. Browse if you want the original framing.