Fill in the blanks: query expansion / query rewriting in RAG and what it costs.

Fill in blank·Medium·4.0 · 0·~1 min·Relevant atPerplexity

Attempt it

Query expansion (also called query rewriting) is a pre-retrieval step in which an rewrites or generates multiple variants of the user's query before retrieval runs. The rewrites are then used to fetch a broader candidate pool, which is fused (often using ) to produce the final top-k. This raises retrieval but adds at least one extra LLM call per query, which costs and on the hot path.

Quick reference

TL;DR

Query expansion uses an LLM to generate multiple query variants, retrieves on each, then fuses results with RRF. Lifts recall on paraphrase heavy queries but adds an extra LLM call's latency and tokens.

Memory aid

Easy to grasp

Imagine asking five friends to phrase the same question in their own words before you send it to a search engine. One friend asks plainly, one friend uses formal vocabulary, one friend translates the question into industry jargon, one friend adds related sub-questions, and one friend strips it down to keywords. You then run all five queries, see what each returns, and merge the lists by giving high ranked items from each list extra credit. You end up with a candidate set that is broader and less likely to miss the right document because of one unlucky phrasing. The catch is you paid a small AI assistant to invent those five variants, which means extra time and extra cost on every query.

Key concepts

RAG system design Embeddings Prompt engineering

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define query expansion, explain why an LLM generates the variants, walk through RRF fusion of N rankings, and contrast against HyDE; close with the latency-cost tradeoff and when routing beats global enablement.

Code example

python

from collections import defaultdict

EXPANSION_PROMPT = """Generate {n} alternate phrasings of the user's question.
Vary style: formal, keyword only, sub-question decomposition.
Return one per line.
User question: {q}"""

def expand_query(q, n=4, small_llm="claude-haiku-4.7"):
    variants = call_llm(small_llm, EXPANSION_PROMPT.format(n=n, q=q))
    return [q] + variants.splitlines()  # always include the original

def rrf(rankings, k=60, top_k=20):
    scores = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] += 1.0 / (k + rank + 1)
    return [d for d, _ in sorted(scores.items(), key=lambda kv: kv[1], reverse=True)[:top_k]]

def retrieve_with_expansion(query, index, top_k=20):
    variants = expand_query(query, n=4)
    rankings = [index.search(embed(v), top_k=top_k) for v in variants]
    return rrf(rankings, k=60, top_k=top_k)

Side-by-side

Variant style	What it captures	When it helps
Formal rephrasing	Corpus written in formal register	User asks casually, docs are formal
Keyword-only version	Lexical hits BM25 would catch	Hybrid retrieval, rare-token queries
Sub-question decomposition	Multi-hop or compound queries	Complex questions that mix facts
HyDE hypothetical answer	Answer-document semantic alignment	Long-document corpora, narrative QA
Multilingual rephrasing	Cross-lingual retrieval	Multilingual corpora

Real-world examples

Real products, models, and research that use this idea.

LangChain's `MultiQueryRetriever` and LlamaIndex's `QueryFusionRetriever` ship the canonical query-expansion-plus-RRF pattern out of the box.
HyDE (Hypothetical Document Embeddings) is a query-expansion variant where the LLM writes a hypothetical answer to the query; the answer is embedded and used as the retrieval query, exploiting answer document semantic alignment.
Claude Haiku 4.7, GPT-5.5-mini, and Gemini 3.1 Flash are the 2026 references for the small fast LLM that generates the variants without dominating latency.
Pinecone, Weaviate, and Qdrant all support running multiple parallel queries against the same index, which is the natural execution model for expanded queries.
Voyage rerank-2 and Cohere Rerank are commonly applied on top of the fused candidate pool to recover precision after expansion has lifted recall.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does HyDE differ from straightforward paraphrasing style query expansion?

QHow would you decide whether query expansion is worth shipping?

QWhat is the right number of variants and why?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Adding query expansion without measuring whether the recall lift on your eval set justifies the per-query latency and cost. Many corpora gain almost nothing from rewriting.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.