Query expansion (also called query rewriting) is a pre-retrieval step in which an rewrites or generates multiple variants of the user's query before retrieval runs. The rewrites are then used to fetch a broader candidate pool, which is fused (often using ) to produce the final top-k. This raises retrieval but adds at least one extra LLM call per query, which costs and on the hot path.
Query expansion is a pre-retrieval step that improves recall when users phrase questions in ways that do not match the corpus's wording.
Instead of embedding the raw user query and searching once, you first call an **LLM** to generate several alternate phrasings of the same question. Common variants include a more formal version, a keyword only version, a sub-question decomposition, and sometimes a hypothetical answer (the HyDE pattern). You then run retrieval once per variant and combine the result lists.
The combination is usually Reciprocal Rank Fusion. RRF scores each candidate by `1 / (k + rank)` summed across all the variant rankings, then picks the top items. It is rank based and needs no score calibration, which makes it the standard fusion choice when combining N independent rankings.
The upside is higher recall on paraphrase heavy or under-specified queries. The downside is one extra LLM call per request, which costs tokens and adds latency. Production teams gate query expansion on workloads where the recall lift earns the extra cost.
Query expansion is the standard pre-retrieval intervention when a user's phrasing is too different from the corpus vocabulary to retrieve well.
**The mechanism is straightforward**. A small fast LLM (Claude Haiku 4.7, GPT-5.5-mini, Gemini 3.1 Flash) receives the user query and a prompt that asks for variants. The variants include alternate phrasings, sub-questions, keyword versions, or hypothetical answers. Each variant is then embedded and used to query the vector index independently. The result is N ranked candidate lists.
Fusion combines those lists into a single final ranking. **Reciprocal Rank Fusion** is the default because it uses only ranks, not scores. The fused score of a candidate is `sum over variants of 1 / (k + rank)`, typically with k around 60. Candidates that rank highly in multiple variants surface to the top; candidates that appear in only one are still represented.
The benefit is a recall lift, especially on paraphrase heavy traffic, under-specified queries, and multilingual workloads. The cost is one or more extra LLM calls per query, which is latency on the hot path and tokens on the bill. Production teams measure recall lift against added latency and cost, and gate the feature behind workloads where it pays off.
Quick reference
TL;DR
Query expansion uses an LLM to generate multiple query variants, retrieves on each, then fuses results with RRF. Lifts recall on paraphrase heavy queries but adds an extra LLM call's latency and tokens.
Imagine asking five friends to phrase the same question in their own words before you send it to a search engine. One friend asks plainly, one friend uses formal vocabulary, one friend translates the question into industry jargon, one friend adds related sub-questions, and one friend strips it down to keywords. You then run all five queries, see what each returns, and merge the lists by giving high ranked items from each list extra credit. You end up with a candidate set that is broader and less likely to miss the right document because of one unlucky phrasing. The catch is you paid a small AI assistant to invent those five variants, which means extra time and extra cost on every query.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Corpus and user vocabulary already align (FAQ corpora, internal docs) No lexical semantic gap means no recall to gain; expansion adds latency and cost with no measurable lift.
Hard sub-second latency budget on the hot path One LLM call adds 100-500ms; that may exceed the entire deadline for real-time agents or interactive search as you type.
Hybrid retrieval already covers the paraphrase gap BM25 plus dense fusion often covers what expansion would cover. Stack only if a stratified eval shows residual recall gaps.
Quick scan · 2–4 min
Everything important, quickly.
3 min: define query expansion, explain why an LLM generates the variants, walk through RRF fusion of N rankings, and contrast against HyDE; close with the latency-cost tradeoff and when routing beats global enablement.
Code example
python
from collections import defaultdict
EXPANSION_PROMPT = """Generate {n} alternate phrasings of the user's question.
Vary style: formal, keyword only, sub-question decomposition.
Return one per line.
User question: {q}"""
def expand_query(q, n=4, small_llm="claude-haiku-4.7"):
variants = call_llm(small_llm, EXPANSION_PROMPT.format(n=n, q=q))
return [q] + variants.splitlines() # always include the original
def rrf(rankings, k=60, top_k=20):
scores = defaultdict(float)
for ranking in rankings:
for rank, doc_id in enumerate(ranking):
scores[doc_id] += 1.0 / (k + rank + 1)
return [d for d, _ in sorted(scores.items(), key=lambda kv: kv[1], reverse=True)[:top_k]]
def retrieve_with_expansion(query, index, top_k=20):
variants = expand_query(query, n=4)
rankings = [index.search(embed(v), top_k=top_k) for v in variants]
return rrf(rankings, k=60, top_k=top_k)
Side-by-side
Variant style
What it captures
When it helps
Formal rephrasing
Corpus written in formal register
User asks casually, docs are formal
Keyword-only version
Lexical hits BM25 would catch
Hybrid retrieval, rare-token queries
Sub-question decomposition
Multi-hop or compound queries
Complex questions that mix facts
HyDE hypothetical answer
Answer-document semantic alignment
Long-document corpora, narrative QA
Multilingual rephrasing
Cross-lingual retrieval
Multilingual corpora
Real-world examples
Real products, models, and research that use this idea.
1LangChain's `MultiQueryRetriever` and LlamaIndex's `QueryFusionRetriever` ship the canonical query-expansion-plus-RRF pattern out of the box.
2HyDE (Hypothetical Document Embeddings) is a query-expansion variant where the LLM writes a hypothetical answer to the query; the answer is embedded and used as the retrieval query, exploiting answer document semantic alignment.
3Claude Haiku 4.7, GPT-5.5-mini, and Gemini 3.1 Flash are the 2026 references for the small fast LLM that generates the variants without dominating latency.
4Pinecone, Weaviate, and Qdrant all support running multiple parallel queries against the same index, which is the natural execution model for expanded queries.
5Voyage rerank-2 and Cohere Rerank are commonly applied on top of the fused candidate pool to recover precision after expansion has lifted recall.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does HyDE differ from straightforward paraphrasing style query expansion?
Sign in or go Pro to see the approach for this follow-up.
QHow would you decide whether query expansion is worth shipping?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right number of variants and why?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Adding query expansion without measuring whether the recall lift on your eval set justifies the per-query latency and cost. Many corpora gain almost nothing from rewriting.