How would you use signals from live production traffic to make a RAG system's retrieval better over time? Name the signals, what you turn them into, and how they feed back into the retriever.
Turn clicks, thumbs and retries into labeled pairs, mine hard negatives, then fine-tune the embedder or train a reranker — adapting retrieval to your real query distribution, with debiasing and A/B tests as guardrails.
Imagine a librarian who hands you three books for every question. Over weeks, she watches which book you actually open, which you flip shut in disgust, and which questions make you come back and ask again differently. She writes all that down. Then she retrains herself: for this kind of question, that book was the right one, and the book that looked promising but you ignored was a sneaky wrong match worth remembering. That is relevance feedback in RAG. The clicks, the thumbs-down, the re-asks become a study guide built from your real readers, so her recommendations get better at exactly the questions your community asks — not generic ones from the whole internet.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A RAG system that ships and then never improves is leaving its best training data on the floor. Every query a real user types, every chunk they click or ignore, every thumbs-down and frustrated rephrase is a labeled judgment about whether retrieval did its job — generated for free, in your exact domain, against your exact query distribution.
The interview wants to see whether you can turn that exhaust into a disciplined learning loop. The naive version — "log thumbs-ups and retrain on them" — sounds reasonable and quietly poisons the system. The strong version names the right signals, mines the right negatives, picks the cheaper feedback path first, and treats the loop's own bias as the central risk.
This deep dive walks through the signals, the labeling step where hard negatives matter most, the two ways to feed the labels back, and the debiasing and validation discipline that keeps the loop from amplifying itself.
The signals live traffic hands you for free
Start by inventorying what production already emits. The signals split into two kinds.
Implicit signals come from behavior. Which retrieved chunk or citation did the user click? How long did they dwell on the answer before moving on? Did they copy or accept the answer, which suggests it landed? Or did they immediately rephrase the same question and retry, which suggests retrieval missed? Downstream conversions — a resolved ticket, a completed purchase — are the strongest implicit signal because they tie back to a real outcome.
Explicit signals come from deliberate user action: thumbs up or down, a star rating, a "this didn't help" button.
The key mindset is that every one of these is a noisy proxy, not ground truth. A click might mean relevance, or it might mean the result was on top. A thumbs-down might mean bad retrieval, or a bad generation on top of good retrieval. The engineering work is converting noisy proxies into labels you can trust, which is why no single signal is taken at face value — you aggregate across sessions and filter outliers.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Search teams build learning to rank rerankers from click logs, applying click models to debias for position before treating clicks as relevance.
- RAG teams fine-tune embedding models (e.g. via sentence-transformers contrastive training) on mined positive pairs and hard negatives drawn from their own query logs.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is a click on the top result a biased relevance signal, and how do you correct it?
Top results get clicked partly because of position, not relevance; use a click model or inverse-propensity weighting to discount position before treating clicks as labels.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Training only on positives — the chunks users liked — and skipping hard negatives: high-ranked chunks the user ignored, which teach the model far more than random unrelated documents.
60 second bullets to scan on the way to the call.
The implicit and explicit relevance signals available in production traffic
How to turn raw signals into labeled query-document pairs
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.