Defend computing online retrieval recall from production traces without a labeled ground truth

Short answer·Hard·4.0 · 0·~3 min·Asked atGoldman Sachs Oracle Voyage Ai

Attempt it

You want a continuous signal of retrieval recall in production but you do not have labeled ground truth. Defend a method that uses traces alone to approximate it, and acknowledge what the method cannot tell you.

Free · 2 AI evals / day

Without labeled ground truth, you cannot compute classical retrieval recall (the fraction of relevant documents that were retrieved). From traces alone you can compute a **proxy** that tracks the same health. The method is **cited-chunk recall**: 1. Prompt the model to cite the chunks it used (e.g. `[chunk_3, chunk_7]`). 2. Record the retrieved chunk ids and the cited chunk ids (parsed from the model output) as span attributes. 3. Per trace, compute `recall_proxy = num_cited / num_retrieved`. 4. Aggregate over time: rolling median, p25, distribution by route. **Why it works.** When the metric is high (say 50 to 80 percent of retrieved chunks get cited), the model is using retrieval. A drop means the model is ignoring retrieval (instruction drift) or the retrieved chunks are too off-topic to cite honestly. **The blind spot.** The metric cannot detect a correct chunk missing from the retrieval set. The model cannot cite what was never retrieved, so cited-chunk recall stays high even when real recall is poor. To cover it, pair the online proxy with a **periodic offline eval against a labeled golden set** (say weekly). The online proxy is a leading indicator; the offline check catches chunks missing from the index.

**Cited-chunk recall is the right online proxy** for retrieval recall when you do not have labeled ground truth. **Construction:** - Prompt the generation step to emit explicit citation markers (structured JSON `citations: [ids]` is cleanest). - Retrieval span records `retrieved_ids`. Generation span parses and records `cited_ids`. - Per-trace metric: `recall_proxy = |cited_ids ∩ retrieved_ids| / |retrieved_ids|`. - Aggregate: rolling p50 and p25 grouped by route and top_k bucket. **Why it is a useful proxy.** The metric correlates with retrieval health in three regimes: - Retrieval brings back relevant chunks and the model cites them: proxy is high. - Retrieval brings back off-topic chunks, so the model cannot cite honestly: proxy drops. - The model is ignoring retrieval (drift, degradation): proxy also drops. A single number tracks two failure modes. A drop is a leading indicator that something regressed. **The blind spot.** The proxy cannot detect cases where the correct chunk was missing from the corpus or the top_k. The model cannot cite what was never retrieved. The proxy can be 80 percent on a query whose real recall is 20 percent. **The offline complement.** Pair the online proxy with a weekly offline eval against a 100 to 500 example golden set to compute real recall@k. The online proxy is the continuous leading indicator. The offline replay is the ground-truth reference. **A second proxy worth tracking** is an LLM-as-judge **answer-grounded score**: does the answer claim only what the retrieved chunks support. Both are cheap to add, with different blind spots.

TL;DR

Compute cited-chunk recall (chunks cited by answer divided by chunks retrieved) per trace, aggregated as a continuous online signal. Pair it with an offline golden-set eval to catch chunks missing from the index.

Memory aid

Easy to grasp

Imagine a student writing an essay using a stack of source books. You cannot read every book yourself, but you can count how many books the student footnoted versus how many sat unused on the desk. If most of the stack gets cited, the books were probably useful. If barely any get cited, either the books were off-topic or the student is ignoring them. What you cannot see is whether the right book was missing from the desk in the first place. So once in a while you check against a small known-good reading list to catch that. The everyday footnote-counting is your live signal. The periodic check is your safety net.

Key concepts

Observability & Tracing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Online retrieval recall cannot be computed in production without labeled ground truth, but there is a best proxy. Cited-chunk recall is the proxy production RAG teams converged on. It computes a per-trace ratio that correlates with retrieval health and gives a useful continuous signal. It has a known blind spot that you cover with a periodic offline check.

Mental model: without labels you cannot measure real recall. You can measure whether the answer leans on retrieval (cited-chunk ratio). The proxy catches some failure modes and is blind to one. Pair it honestly with a periodic offline check.

Construction: from prompt discipline to span attribute

Citation discipline in the prompt

The generation step must be prompted to cite. Two viable formats:

Inline markers: 'According to [chunk_3], the policy is to refund within 30 days.'
Structured JSON: { "answer": "...", "citations": ["chunk_3", "chunk_7"] }.

The structured form is easier to parse and harder for the model to skip. Prefer it.

The prompt should also instruct: 'cite every key claim; if no retrieved chunk supports a claim, say so or omit the claim.' This keeps the model from inventing citations.

Span attributes

Retrieval span:

retrieval.retrieved_ids: ordered list of chunk ids.
retrieval.scores: parallel list of relevance scores.
retrieval.top_k: the K used.

Generation span:

generation.cited_ids: chunk ids extracted from the model output.
generation.num_citations: convenience count.

The extraction step is deterministic: parse the JSON output, extract the citations field. If the model output is non-conformant (free text instead of JSON), record num_citations = 0 and emit a flag so you can track conformance separately.

Per-trace metric

\text{recall\_proxy} = \frac{|\text{cited\_ids} \cap \text{retrieved\_ids}|}{|\text{retrieved\_ids}|}

The intersection step is important. A citation that does not match any retrieved id is a hallucinated citation, not a real one. It should not count toward recall.

Aggregation

Compute rolling p50 and p25 over 1, 6, 24 hour windows. Group by route (different routes have different retrieval configurations), top_k bucket (larger top_k tends to lower the proxy), model version, retriever version.

What the proxy catches and what it cannot

The offline golden-set complement

The four-quadrant diagnostic with judge groundedness

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

In Langfuse, citation coverage is not a default metric. It can be logged per trace as a custom score.
Arize Phoenix ships a DocumentRelevanceEvaluator that operates on retrieved chunks and is the natural pair to citation-coverage.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you weight cited-chunk recall by retrieval rank?

Each retrieved chunk has a rank. A weighted proxy: sum of inverse-rank for cited chunks, normalized by sum of inverse-rank for retrieved chunks. High-rank cites count more; deep cites count less. This catches the case where the model only cites the easy top-1.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating cited-chunk recall as real recall. It cannot detect when the right chunk was never in the retrieved set. Pair it with an offline golden-set eval.

60 second bullets to scan on the way to the call.

What cited-chunk recall actually measures
Why it is a proxy and not real recall

Primary sources. Browse if you want the original framing.