Knowledge lives in a SQL database, not documents: how does the RAG approach change?

The instinct with RAG is to embed everything and do similarity search, but for a relational database that is the wrong tool. The questions people ask of structured data are usually **exact and aggregative**: "total revenue in Q3," "how many orders over 500 dollars." These need precise computation — sums, filters, joins, counts — over the rows. Vector similarity cannot do any of that. The nearest neighbors are rows that look semantically similar to the question, not a correct result. There is no way to sum a column or join two tables with a nearest-neighbor lookup. The right pattern is **text to SQL**. The LLM translates the question into a SQL query, the database executes it, and you return or summarize the rows it produces. The exact computation is done by the engine, which is what databases are good at. Notice what "retrieval" means here. You do not retrieve the data — you retrieve the schema and a few example question to SQL pairs, so the model can write a correct query. The hard parts are keeping the SQL correct and safe: read-only access, validating the query, and feeding execution errors back so the model can fix its mistakes.

Naive vector retrieval fails here because of a mismatch between the questions and the mechanism. Questions over a relational store — sales, inventory, user records — are typically **exact and aggregative**: totals, counts, filters by precise thresholds, joins across tables, group-bys by period. Embedding the rows and doing nearest-neighbor search returns rows semantically close to the query string, not the right answer. You cannot sum a revenue column or join customers to orders with cosine similarity. The correct RAG pattern is **text to SQL**: the model generates a SQL query from the question, the database executes it, and the precise rows are returned or summarized. The exact computation lives where it belongs — in the query engine — and the model's job is translation, not arithmetic. The key reframe is what gets retrieved. Retrieval is no longer over the data; it is over the schema and exemplars. You fetch the relevant table and column definitions and a few example question to SQL pairs, so the model writes a query that matches your actual schema. The "retrieval" of the answer itself is then a deterministic database query, not an approximate similarity match. The hard parts are correctness and safety. The model can hallucinate column names or write a wrong join, so you want schema grounding, query validation, and an error-repair loop that feeds execution errors back for self-correction. And because you are executing model-written SQL, you enforce read-only access and guard against expensive or injection-style queries.