Defend computing online retrieval recall from production traces without a labeled ground truth

Short answer·Hard·4.0 · 0·~3 min·Asked atGoldman Sachs Oracle Voyage Ai

Attempt it

You want a continuous signal of retrieval recall in production but you do not have labeled ground truth. Defend a method that uses traces alone to approximate it, and acknowledge what the method cannot tell you.

Free · 2 AI evals / day

Without labeled ground truth, you cannot compute classical retrieval recall (the fraction of relevant documents that were retrieved). What you can compute from traces alone is a **proxy** that tracks the same underlying health. The method is **cited-chunk recall**: 1. Prompt the generation step to emit explicit citations to the chunks it used (e.g. `[chunk_3, chunk_7]`). 2. Capture both the retrieved chunk ids (from the retrieval span) and the cited chunk ids (parsed from the model output) as span attributes. 3. Per trace, compute `recall_proxy = num_cited / num_retrieved`. 4. Aggregate over time: rolling median, p25, distribution by route. **Why this is a useful proxy.** When the metric is high (say 50 to 80 percent of retrieved chunks get cited), the model is using retrieval. When it drops, one of two things is happening: the model is ignoring retrieval (instruction drift), or the retrieved chunks are off-topic enough that the model cannot cite them honestly. Both are real problems you want to know about. **The blind spot.** The metric cannot detect the case where the correct chunk was missing from the retrieval set entirely. The model cannot cite something that was never retrieved, so cited-chunk recall stays high even when real recall is poor. To cover the blind spot, pair the online proxy with a **periodic offline eval against a labeled golden set** (say weekly). Online: continuous proxy as a leading indicator. Offline: periodic ground-truth check that catches missing from index regressions.

**Cited-chunk recall is the right online proxy** for retrieval recall when you do not have labeled ground truth. **Construction:** - Prompt the generation step to emit explicit citation markers (structured JSON `citations: [ids]` is cleanest). - Retrieval span records `retrieved_ids`; generation span parses and records `cited_ids`. - Per-trace metric: `recall_proxy = |cited_ids ∩ retrieved_ids| / |retrieved_ids|`. - Aggregate: rolling p50 and p25 grouped by route and top_k bucket. **Why it is a useful proxy.** The metric correlates with retrieval health in two regimes: - Retrieval brings back relevant chunks, the model cites them, proxy is high. - Retrieval brings back off-topic chunks, the model cannot cite honestly, proxy drops. - The model is ignoring retrieval (drift, degradation), proxy also drops. A single number tracks two failure modes. A drop is a leading indicator that something regressed. **The honest blind spot.** The proxy cannot detect cases where the correct chunk was missing from the corpus or missing from the top_k. The model cannot cite what was never retrieved. The proxy can be 80 percent on a query whose real recall is 20 percent. **The offline complement.** Pair the online proxy with a weekly offline eval against a 100 to 500 example golden set; compute real recall@k. The online proxy is the continuous leading indicator; the offline replay is the ground-truth anchor. **A second proxy worth tracking** alongside cited-chunk recall is an LLM-as-judge **answer-grounded score**: does the answer claim only what the retrieved chunks support. Different blind spots, both cheap to add.

TL;DR

Compute cited-chunk recall (chunks cited by answer divided by chunks retrieved) per trace; aggregate as a continuous online signal. Pair with offline golden-set eval to cover the missing from index blind spot.

Memory aid

Easy to grasp

Imagine a student writing an essay using a stack of source books. You cannot read every book yourself, but you can count how many books the student footnoted versus how many sat unused on the desk. If most of the stack gets cited, the books were probably useful. If barely any get cited, either the books were off-topic or the student is ignoring them. What you cannot see is whether the right book was missing from the desk in the first place. So once in a while you check against a small known-good reading list to catch that. The everyday footnote-counting is your live signal; the periodic check is your safety net.

Key concepts

Observability & Tracing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Computing online retrieval recall in production without labeled ground truth is one of those problems where the honest answer is 'you cannot, but here is the best proxy.' Cited-chunk recall is the proxy production RAG teams converged on between 2024 and 2026. It computes a per-trace ratio that correlates with retrieval health, surfaces a useful continuous signal, and has a known blind spot that you cover with a periodic offline check.

This walkthrough explains the metric construction, the two failure modes it catches, the one structural failure mode it cannot catch, the offline complement that closes the gap, and the four-quadrant diagnostic that emerges when you pair cited-chunk recall with an answer-grounded judge score.

Mental model: without labels you cannot measure real recall. You can measure whether the answer leans on retrieval (cited-chunk ratio). The proxy catches some failure modes and is blind to one. Pair it honestly with a periodic offline check.

Construction: from prompt discipline to span attribute

Citation discipline in the prompt

The generation step must be prompted to cite. Two viable formats:

Inline markers: 'According to [chunk_3], the policy is to refund within 30 days.'
Structured JSON: { "answer": "...", "citations": ["chunk_3", "chunk_7"] }.

The structured form is easier to parse and harder for the model to skip. Prefer it.

The prompt should also instruct: 'cite every load-bearing claim; if no retrieved chunk supports a claim, say so or omit the claim.' This keeps the model from inventing citations.

Span attributes

Retrieval span:

retrieval.retrieved_ids: ordered list of chunk ids.
retrieval.scores: parallel list of relevance scores.
retrieval.top_k: the K used.

Generation span:

generation.cited_ids: chunk ids extracted from the model output.
generation.num_citations: convenience count.

The extraction step is deterministic: parse the JSON output, extract the citations field. If the model output is non-conformant (free text instead of JSON), record num_citations = 0 and emit a flag so you can track conformance separately.

Per-trace metric

\text{recall\_proxy} = \frac{|\text{cited\_ids} \cap \text{retrieved\_ids}|}{|\text{retrieved\_ids}|}

The intersection step is important: a citation that does not match any retrieved id is a hallucinated citation, not a real one, and should not count toward recall.

Aggregation

Rolling p50 and p25 over 1, 6, 24 hour windows. Group by route (different routes have different retrieval configurations), top_k bucket (larger top_k tends to drag the proxy down), model version, retriever version.

What the proxy catches and what it cannot

The offline golden-set complement

The four-quadrant diagnostic with judge groundedness

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Langfuse 3.x's RAG dashboards surface citation-coverage as a default metric for retrieval-augmented apps.
Arize Phoenix 5.x ships a RelevanceEvaluator that operates on retrieved chunks and is the natural pair to citation-coverage.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you weight cited-chunk recall by retrieval rank?

Each retrieved chunk has a rank. A weighted proxy: sum of inverse-rank for cited chunks, normalized by sum of inverse-rank for retrieved chunks. High-rank cites count more; deep cites count less. Catches the case where the model only cites the easy top-1.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating cited-chunk recall as real recall. It cannot detect when the right chunk was never in the retrieved set; pair with offline golden-set eval.

60 second bullets to scan on the way to the call.

What cited-chunk recall actually measures
Why it is a proxy and not real recall

Primary sources. Browse if you want the original framing.