Similarity search returns approximately relevant text, not the exact rows an aggregate needs; route numeric and aggregate questions to structured retrieval like text to SQL so totals are computed exactly.
Imagine asking a helper to total your monthly spending, but instead of handing them your bank statement, you hand them a stack of sticky notes that merely look related to money. They'll add up whatever notes they happened to grab and confidently give you a number — but it'll be wrong, because some receipts were never in the stack and they're doing the math in their head. The fix isn't a thicker stack of sticky notes or fancier handwriting. The fix is to pull up the actual statement in a spreadsheet and let the calculator add the column. For a system reading financial reports, that means turning 'what's the total revenue?' into a precise database query that runs over the real rows, instead of letting the language model eyeball a pile of similar-looking text.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A finance team builds RAG over their reports, demos it on 'what does this footnote mean,' and it's great. Then someone asks 'what was total revenue across all regions last quarter,' and it returns a confident, wrong number. The instinct is to treat this as a retrieval bug and start turning the usual knobs.
That instinct is the trap this question sets. The failure isn't that retrieval is slightly off; it's that retrieval is the wrong mechanism for the request. An aggregate is a computation over a known set of records, and vector search neither guarantees the complete set nor performs the computation. Understanding why all the tempting retrieval fixes fail — and why the answer is to change the architecture rather than tune it — is the difference between a junior and a senior read on RAG over structured data.
The stakes are higher than usual because the output is a number someone may act on. A vague summary that's slightly off is forgivable; a revenue total that's wrong by 60 million is a reporting error. And the system gives no warning — it returns a clean, formatted figure with full confidence. That combination, high stakes plus silent failure, is exactly why the architecture has to guarantee correctness rather than hope retrieval got lucky.
What vector search actually returns, and why that breaks aggregation
Dense retrieval embeds the query and finds chunks whose vectors are closest by cosine similarity:
The operative word is closest. The result is the set of passages that are approximately most similar to the question. That's exactly right for 'find the relevant text,' and exactly wrong for 'compute a value over a defined set.'
Think about 'sum revenue across all segments.' A correct answer requires every segment's figure, no more and no fewer. Similarity search has no notion of completeness. It returns the top-k most similar chunks, and if a segment's number lives in a passage that didn't rank in the top-k — maybe it's phrased differently, maybe it's in a table the embedder handled poorly — that figure silently never reaches the model. The sum is then computed over a partial set.
Worse, the system has no error signal. A SQL SUM over the wrong WHERE clause at least runs; a similarity search that missed a row looks identical to one that found everything. The retriever did its job — return similar text — and that job was never aggregation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Question type | Right retrieval approach |
|---|---|
| Exact figures or aggregates (sum, count, max, compare) | Structured query: text to SQL or table extraction, executed for an exact result |
| Fuzzy or semantic lookup over prose | Vector retrieval over chunks |
| Mixed (narrative plus a number) | Route per sub-question, then combine the results |
Real products, models, and research that use this idea.
- A finance assistant over 10-K filings that quoted wrong segment totals until aggregate questions were routed to text to SQL over an extracted tables database.
- LlamaIndex and LangChain ship SQL query engines and table-retrieval tools precisely for this numeric-question path alongside vector retrieval.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the router decide a question is numeric/aggregate versus a factual lookup?
Use an LLM or lightweight classifier on the query intent, often keyed on aggregate cues (total, average, count, compare, trend) and on whether the answer is a computed quantity over records. Route accordingly; some systems try both paths and reconcile.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Trying to fix wrong totals by raising top-k or shrinking chunks; both keep the LLM doing unreliable mental math over an incomplete set of figures.
60 second bullets to scan on the way to the call.
Why similarity search returns an approximate, possibly incomplete set of figures
Why LLMs are unreliable at arithmetic over scattered chunks
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.