Spot the bug: a junior engineer's conversational RAG implementation produces nonsense retrievals on turn 3+. Find the error in the retrieval logic.

Spot the error·Medium·4.0 · 0·~2 min·Relevant atAnthropic Perplexity

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

The bug is on the very first line of the function. It embeds the raw `new_user_message` and uses that as the search query. This works for turn 1, where the user's message is self-contained, and fails immediately afterward. In a real conversation, users do not repeat the full context every turn. Turn 1 might be 'what does the Pro plan include?'. Turn 2 might be 'and what about Enterprise?'. Turn 3 might be 'how does it compare?'. Embedding 'and what about Enterprise?' alone gives the embedding model no idea what the user means by 'what about'; it might be plans, weather, dinner, or anything else. The retriever then returns chunks that match the surface words, which are almost never what the user meant. The fix is a **query rewriting** step before embedding. Call a small fast LLM with the conversation history plus the new message, and ask it to produce a standalone, self-contained query. Embed that rewritten query. The cost is one extra LLM call (100-300ms with Claude Haiku 4.7 or GPT-5.5-mini), and without it conversational RAG quality drops off a cliff after the first turn.

The bug is that the function embeds `new_user_message` directly, ignoring the conversation history. This breaks the moment the user uses coreference or ellipsis. That happens within two or three turns of any real conversation. **Why turn 1 works**. The user's first message is almost always self-contained. 'What does the Pro plan include?' has a noun phrase ('the Pro plan'), a verb ('include'), and no pronouns. The embedding model encodes the meaning cleanly. The retriever returns the right chunks. **Why turn 3 collapses.** By turn 3 users say 'and what about Enterprise?' or 'how does it compare?'. These messages are perfectly clear to a human reading the conversation, because the pronouns and elliptical references resolve through prior turns. They are useless to an embedding model that sees only the raw string. The vector encodes the surface form, not the resolved meaning. The retriever surfaces chunks about 'comparison', 'enterprise', or 'cost' in general. It misses chunks about the specific subject from earlier turns. **The fix is query rewriting.** Insert a pre-retrieval step. It calls a small fast LLM (Claude Haiku 4.7, GPT-5.5-mini, Gemini 3.1 Flash) with the conversation history plus the new user message. Ask it to produce a standalone, self-contained query. Embed the rewritten query. The cost is one LLM call per turn (typically 100-300ms). Without it, conversational RAG quality collapses by turn 3.

Quick reference

TL;DR

Embedding the raw new_user_message ignores conversation context, so coreference like 'it' or 'that' makes the retrieval query meaningless. Fix: rewrite to a standalone query using history before embedding.

Memory aid

Easy to grasp

Imagine handing someone a sticky note that just says 'what does it cost?' and asking them to find the answer in a library. They have no idea what 'it' refers to. To do their job, they need the previous notes from the same conversation. That is exactly what your retriever needs in a multi-turn chat. Turn one usually contains a full question, so the retriever does fine. By turn three, users naturally drop the subject and use pronouns: 'how about for enterprise?', 'what does it cost?', 'what about the others?'. Before searching, you need to glue the new message together with what came before and rewrite it into a standalone question. Now the retriever sees something it can actually search on.

Key concepts

RAG system design Embeddings Prompt engineering

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: trace a 3-turn conversation, show why the raw new_user_message embedding fails at turn 3, name the rewrite step and which model handles it, then describe the recall by turn stratification that catches the bug.

Code example

python

REWRITE_PROMPT = """Given the conversation so far and the user's new message,
produce a standalone search query that resolves any pronouns or elliptical references.
If the new message is already self-contained, return it unchanged.

Conversation:
{history}

New message: {message}

Standalone query:"""

def retrieve_chunks(conversation_history, new_user_message, vector_store, top_k=5,
                   rewriter_llm="claude-haiku-4.7"):
    # 1. Rewrite to a standalone query.
    if conversation_history:
        rewritten = call_llm(
            rewriter_llm,
            REWRITE_PROMPT.format(
                history=format_history(conversation_history),
                message=new_user_message,
            ),
        ).strip()
    else:
        rewritten = new_user_message  # first turn, nothing to rewrite

    # 2. Embed the standalone query, not the raw user message.
    query_embedding = embed(rewritten)
    return vector_store.search(query_embedding, top_k=top_k)

Side-by-side

Conversation turn	Raw message	Standalone rewrite	Retrieval outcome without rewrite
Turn 1	'What does the Pro plan include?'	Same (already standalone)	Good
Turn 2	'And what about Enterprise?'	'What does the Enterprise plan include?'	Partial; surface 'enterprise' content but not pricing context
Turn 3	'How does it compare?'	'How does the Enterprise plan compare to the Pro plan?'	Bad; retrieves generic 'comparison' content
Turn 4	'What does it cost?'	'What does the Enterprise plan cost?'	Bad; retrieves arbitrary pricing content

Real-world examples

Real products, models, and research that use this idea.

LangChain's `condense_question_prompt` in `ConversationalRetrievalChain` is the canonical implementation of conversational query rewriting; it takes history plus new question and emits a standalone reformulation.
LlamaIndex's `CondenseQuestionChatEngine` and `CondensePlusContextChatEngine` ship the same pattern with slightly different orchestration.
Anthropic Claude Haiku 4.7 and OpenAI GPT-5.5-mini are the 2026 references for the small fast LLM that does the rewrite step in 100-300ms.
Voyage Embed v3 and OpenAI text-embedding-3-large are commonly paired with this rewrite step; the embedding model is unchanged, just the query that goes into it.
Pinecone, Weaviate, and Qdrant see this pattern in 80%+ of production conversational RAG deployments built on top of them in 2026.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat if the conversation has 30 turns? Do you pass all of them into the rewriter?

QHow would you detect this bug in an existing production system without ground-truth labels?

QCan you skip the rewrite step on turns where the new message is already self-contained?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Embedding the raw new_user_message in a multi-turn chat. Coreference and ellipsis make turn 3+ unintelligible on its own; retrieval quality collapses without a rewrite step.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.