Design query routing for a RAG app serving several distinct knowledge sources

Short answer·Medium·4.0 · 0·~3 min·Asked atLightning Ai Sarvam Stripe

Attempt it

A RAG app must answer over several distinct sources: product documentation, a relational database, and live web results. Design a query routing layer: what it does, why a single index is not enough, and how routing decisions are made.

Free · 2 AI evals / day

**Query routing** is a step that sits in front of retrieval. It reads the incoming query, decides which knowledge source it belongs to, and sends it there with the right retrieval method. Why not one index? Because the three sources have different shapes. Product docs are text you match by meaning — a vector index fits. A relational database answers exact aggregates like 'how many orders shipped last week', which needs text to SQL, not similarity search. Live web results need a real-time search call. Merge them into one index and every query is forced through one retrieval shape that fits none well, so relevance drops. The router maps query types to sources. 'How do I configure feature X?' goes to the docs index. 'How many orders last week?' goes to the database. 'What's the latest on Y?' goes to web search. How does it decide? Common options are an LLM classifier or function-calling selector, a small trained classifier, or comparing the query's embedding to a short description of each source. Good designs add a default route for anything unclear, allow sending an ambiguous query to several sources, and log which route fired so misroutes are easy to spot.

The design move is to insert a **router** before retrieval. It is a component that classifies the incoming query and dispatches it to the right index, tool, or retrieval strategy, rather than running one fixed pipeline for everything. A single undifferentiated index is the thing to argue against, because the three sources answer fundamentally different question shapes. The relational database answers exact aggregations — 'how many orders shipped last week' — a text to SQL problem, not similarity search. No chunk's embedding matches a COUNT over a date range. Product docs are semantic text, well served by a vector index. Live web needs a fresh call at query time. Pour all three into one index and the SQL question retrieves topically-near doc noise, and relevance collapses. The routing decision can be made three ways. An **LLM classifier** or function-calling selector picks a route — flexible and zero-training but adds a model call. A small trained classifier is cheaper once you have labeled traffic. Or you embed a short description of each source and route by similarity to the query — no LLM call, easy to extend. Robustness separates good designs from naive ones. Add a default route so an unmatched query still gets answered. Allow fan-out to multiple sources, merging results, when a query spans sources. And make routes observable so a misroute shows up in traces. The payoff: each source is queried with the method it deserves, and the wrong shape of data never reaches the prompt.

TL;DR

Query routing puts a classifier before retrieval that sends each query to the source it needs — SQL for aggregates, the docs index for how-tos, live web for news — not one blurry merged index.

Memory aid

Easy to grasp

Imagine a help desk with three specialists: an accountant with the books, a manual writer who knows the product, and a researcher who reads today's news. If you dumped all their knowledge into one binder and made every visitor flip through it, they'd wade past irrelevant pages every time. A receptionist fixes that. They hear your question, decide which specialist it's for, and send you straight there. Query routing is that receptionist for a RAG app — it reads the question and dispatches it to the right source, so you get an answer from the expert who actually has it, not a mashup of all three.

Key concepts

RAG LLM System Design Vector Databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Most RAG tutorials assume one corpus and one retriever. Real products rarely have that luxury. A support copilot needs account data from a database, how-to answers from product docs, and 'is it down right now' answers from a live status feed. These are not three topics in one library — they are three different kinds of question with three different right answers.

The instinct of someone who has only built single-index RAG is to merge everything into one vector store and let similarity sort it out. That instinct is the trap this question probes. Similarity search assumes the answer lives in a passage whose meaning is close to the query. That assumption breaks the moment a query's answer is a computation over rows, or a fact that did not exist when the index was built.

Query routing is the architectural response: a decision layer that classifies the query and sends it to the source and method that can actually answer it. This deep dive covers why the single-index approach degrades, how the routing decision is made and at what cost, the robustness patterns that production demands, and where routing shades into agentic retrieval.

Why one index cannot serve mixed sources

The single-index objection is not about scale — it is about modality. A vector index answers exactly one kind of question: 'find me passages whose meaning is close to this query.' That is perfect for product docs, where the answer to 'how do I configure feature X' is a passage that talks about configuring feature X.

Now point that same machinery at a relational database. Ask 'how many orders shipped last week.' The answer is a number produced by a COUNT over a date-filtered table — it exists in no passage. If you chunked the rows and embedded them, the query's embedding would land near rows that mention orders or shipping, but none of them is the answer, because the answer is a computation. Similarity search has no notion of aggregation. You get topically-near noise and a confidently wrong number. The right tool is text to SQL: translate the query to a query, run it, return the exact result.

Live web is a third modality, defined by freshness. 'What's the latest on Y' cannot be answered from any static index, however well-built, because the relevant fact may be hours old. It needs a real-time search call.

Merge all three and you do not get a system that handles all three — you get one that handles none well. Mixing modalities in one index dilutes score distributions, so the cutoff that works for docs is wrong for rows; and every query is forced through the single retrieval shape the index supports. The merge does not add capability; it averages away the strengths of each source.

How the routing decision gets made

The robustness layer juniors skip

From router to agent: the same machinery scaling up

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Routing mechanism	Latency / cost	Best fit
LLM classifier / function calling	One model call; non-deterministic	Long-tail queries, fast to ship, multi-source selection
Small trained classifier	Very low once trained	High-volume head of the query distribution
Embedding similarity to source descriptions	One embed + lookup	Cheap default, easy to add sources by writing a description

Real products, models, and research that use this idea.

LlamaIndex's RouterQueryEngine and selector classes pick among query engines using an LLM selector, and its SubQuestionQueryEngine fans a complex query out across multiple sources.
LangChain routing chains and tool-calling agents let an LLM choose between a SQL tool, a vector retriever, and a web-search tool per query.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you handle a query that legitimately spans two sources, like comparing last week's order volume to a documented limit?

This is a fan-out plus join, not a single pick. Decompose the query into sub-queries, route each to its source (SQL for the volume, the docs index for the limit), retrieve both, then merge the evidence into one context for the generator. LlamaIndex's SubQuestionQueryEngine is the canonical pattern. The router's job becomes decomposition and dispatch, not just classification.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Merging structured rows, doc chunks, and web text into one vector index and hoping similarity sorts it out. A SQL aggregate like 'orders last week' has no good chunk to match, so retrieval returns topically-near noise and the answer is wrong.

60 second bullets to scan on the way to the call.

Define query routing and where it sits relative to retrieval.
Explain why one merged index degrades relevance across mixed source types.

Primary sources. Browse if you want the original framing.