Fill in the blanks: query expansion / query rewriting in RAG and what it costs.
Query expansion uses an LLM to generate multiple query variants, retrieves on each, then fuses results with RRF. Lifts recall on paraphrase heavy queries but adds an extra LLM call's latency and tokens.
Imagine asking five friends to phrase the same question in their own words before you send it to a search engine. One friend asks plainly, one friend uses formal vocabulary, one friend translates the question into industry jargon, one friend adds related sub-questions, and one friend strips it down to keywords. You then run all five queries, see what each returns, and merge the lists by giving high ranked items from each list extra credit. You end up with a candidate set that is broader and less likely to miss the right document because of one unlucky phrasing. The catch is you paid a small AI assistant to invent those five variants, which means extra time and extra cost on every query.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Query expansion sounds like a small pre-retrieval polish step and is actually one of the more under-discussed levers in production RAG. It directly attacks the lexical semantic gap, the failure mode where a perfectly good retriever misses the right document because the user phrased the question in words the corpus does not use. Generating two or three alternate phrasings with a small LLM, retrieving on each, and fusing the rankings can lift recall by 5-15 points on paraphrase heavy workloads.
This deep dive walks through the mechanics: which component generates variants, why RRF is the standard fusion choice, how HyDE differs from straight paraphrasing, what the costs actually are on a real production hot path, and when routing the feature on a per-query basis beats running it globally.
Why query expansion exists: the lexical semantic gap
A single embedding of a user's raw query is a single point in vector space. The right document is the document whose embedding sits closest to that point. If the user phrased the question in vocabulary that does not match the corpus, the point may sit closer to wrong documents than to the right one.
This is a real failure mode. A customer asks 'how do I cancel?' against a knowledge base that uses 'terminate', 'unsubscribe', 'end your plan', or 'discontinue service'. Modern embeddings collapse synonyms partially but not perfectly; the single query embedding still favors documents with the exact verb the user used. Recall drops on this exact class of queries.
Query expansion attacks the gap by sampling the same intent in multiple wordings. Instead of one query point, you have N query points scattered around the intent's vector neighborhood. The chance that at least one variant lands near the right document is much higher than the chance that the original phrasing does.
The sister technique HyDE goes further: it asks the LLM to write a hypothetical answer to the question and uses the answer's embedding as the retrieval query. The intuition is that documents are more like answers than like questions, so the answer embedding lives in a region that better matches the corpus.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from collections import defaultdict
EXPANSION_PROMPT = """Generate {n} alternate phrasings of the user's question.
Vary style: formal, keyword only, sub-question decomposition.
Return one per line.
User question: {q}"""
def expand_query(q, n=4, small_llm="claude-haiku-4.7"):
variants = call_llm(small_llm, EXPANSION_PROMPT.format(n=n, q=q))
return [q] + variants.splitlines() # always include the original
def rrf(rankings, k=60, top_k=20):
scores = defaultdict(float)
for ranking in rankings:
for rank, doc_id in enumerate(ranking):
scores[doc_id] += 1.0 / (k + rank + 1)
return [d for d, _ in sorted(scores.items(), key=lambda kv: kv[1], reverse=True)[:top_k]]
def retrieve_with_expansion(query, index, top_k=20):
variants = expand_query(query, n=4)
rankings = [index.search(embed(v), top_k=top_k) for v in variants]
return rrf(rankings, k=60, top_k=top_k)| Variant style | What it captures | When it helps |
|---|---|---|
| Formal rephrasing | Corpus written in formal register | User asks casually, docs are formal |
| Keyword-only version | Lexical hits BM25 would catch | Hybrid retrieval, rare-token queries |
| Sub-question decomposition | Multi-hop or compound queries | Complex questions that mix facts |
| HyDE hypothetical answer | Answer-document semantic alignment | Long-document corpora, narrative QA |
| Multilingual rephrasing | Cross-lingual retrieval | Multilingual corpora |
Real products, models, and research that use this idea.
- LangChain's `MultiQueryRetriever` and LlamaIndex's `QueryFusionRetriever` ship the canonical query-expansion-plus-RRF pattern out of the box.
- HyDE (Hypothetical Document Embeddings) is a query-expansion variant where the LLM writes a hypothetical answer to the query; the answer is embedded and used as the retrieval query, exploiting answer document semantic alignment.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does HyDE differ from straightforward paraphrasing style query expansion?
HyDE asks the LLM to write a hypothetical answer to the query, then embeds the answer as the retrieval query. It exploits the property that documents are more semantically similar to answers than to questions. Paraphrasing rewrites the query in alternative phrasings; HyDE generates a different artifact entirely.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Adding query expansion without measuring whether the recall lift on your eval set justifies the per-query latency and cost. Many corpora gain almost nothing from rewriting.
60 second bullets to scan on the way to the call.
What query expansion is and where it fits in the RAG pipeline
Why an LLM is the natural choice to generate query variants
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.