RAG over financial reports returns wrong totals when asked to sum or compare figures — what is the right design?

The system fails because it's using the wrong tool for the job. Vector search finds text that is approximately similar to the question. For 'what does this clause mean?' that's perfect. For 'what's the total revenue across all segments?' it's the wrong primitive. Two things break at once. First, similarity search returns the chunks that look most related, not a guaranteed-complete set of every figure you need. Some numbers simply aren't retrieved. Second, even if they were all present, the model is being asked to add them up in its head across separate chunks, and that arithmetic is unreliable. The right design is to detect that the question is numeric or aggregate, then route it to structured retrieval. Instead of feeding chunks to the model, you turn the question into a precise query — text to SQL, or extracting the relevant table — and let the data store compute the exact value. More chunks or smaller chunks don't fix this. An exact total needs an exact query, not approximate similarity plus mental math.

The correct answer is to route numeric and aggregate questions to structured retrieval — text to SQL or table extraction. The values are then computed exactly, rather than asking the LLM to add up scattered chunks. The failure has two independent causes. Vector retrieval returns text that is approximately similar to the query, not the complete, exact set of rows an aggregate requires. A sum over 'all segments' needs every segment's figure; similarity search has no guarantee it surfaced all of them. On top of that, even with the figures present, the model is doing arithmetic over numbers spread across separately retrieved chunks, and LLMs are unreliable at multi-step arithmetic. So you get confidently wrong totals. The other options treat symptoms. Raising top-k just floods the context with more chunks — more noise, no guarantee of completeness, and the model is still doing the math. Smaller chunks improve embedding precision but don't make aggregation exact. A bigger financial-text embedding model improves semantic matching, not the model's arithmetic. The principle: exact aggregates require an exact query. Recognize numeric intent, generate a query against the structured source, execute it deterministically, and let the LLM phrase the result. Similarity search is for finding relevant text, not for computing answers over known records.