Your answers must come from a relational database (sales figures, inventory, user records), not a corpus of text. Explain why naive vector retrieval over the rows is the wrong tool and what the RAG pattern looks like instead.
Free · 2 AI evals / day
The instinct with RAG is to embed everything and do similarity search, but for a relational database that is the wrong tool. The questions people ask of structured data are usually **exact and aggregative**: "total revenue in Q3," "how many orders over 500 dollars." These need precise computation — sums, filters, joins, counts — over the rows.
Vector similarity cannot do any of that. The nearest neighbors are rows that look semantically similar to the question, not a correct result. There is no way to sum a column or join two tables with a nearest-neighbor lookup.
The right pattern is **text to SQL**. The LLM translates the question into a SQL query, the database executes it, and you return or summarize the rows it produces. The exact computation is done by the engine, which is what databases are good at.
Notice what "retrieval" means here. You do not retrieve the data — you retrieve the schema and a few example question to SQL pairs, so the model can write a correct query. The hard parts are keeping the SQL correct and safe: read-only access, validating the query, and feeding execution errors back so the model can fix its mistakes.
Naive vector retrieval fails here because of a mismatch between the questions and the mechanism. Questions over a relational store — sales, inventory, user records — are typically **exact and aggregative**: totals, counts, filters by precise thresholds, joins across tables, group-bys by period. Embedding the rows and doing nearest-neighbor search returns rows semantically close to the query string, not the right answer. You cannot sum a revenue column or join customers to orders with cosine similarity.
The correct RAG pattern is **text to SQL**: the model generates a SQL query from the question, the database executes it, and the precise rows are returned or summarized. The exact computation lives where it belongs — in the query engine — and the model's job is translation, not arithmetic.
The key reframe is what gets retrieved. Retrieval is no longer over the data; it is over the schema and exemplars. You fetch the relevant table and column definitions and a few example question to SQL pairs, so the model writes a query that matches your actual schema. The "retrieval" of the answer itself is then a deterministic database query, not an approximate similarity match.
The hard parts are correctness and safety. The model can hallucinate column names or write a wrong join, so you want schema grounding, query validation, and an error-repair loop that feeds execution errors back for self-correction. And because you are executing model-written SQL, you enforce read-only access and guard against expensive or injection-style queries.
Quick reference
TL;DR
Vector search returns similar rows, but structured questions need exact aggregation. RAG becomes text to SQL: retrieve the schema and example queries, generate SQL, execute it, return the precise result.
Imagine you ask a librarian, 'how much did we earn last month?' If the librarian just hands you the three pages that sound most like your question, you still have to add the numbers up yourself — and you might get the wrong pages. What you actually want is for the librarian to walk to the ledger, run their finger down the right column, and add it up exactly. With a database, the trick is to teach the assistant the shape of the ledger (which tables and columns exist) and a few examples of how to look things up, then have it write the precise instruction the database can run. The database does the exact math; the assistant just translates your plain-English question into that instruction and reads back the answer.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Questions over unstructured free-text columns like support-ticket bodies or reviews When the answer lives in prose inside a cell, semantic search over that text beats SQL, which can only match or pattern-filter strings, not understand them.
Open-ended exploratory questions with no precise computation If the user wants themes or a summary rather than an exact figure, generating a single SQL query is the wrong frame; document RAG or aggregation plus summarization fits better.
Schemas so volatile that exemplars and grounding go stale constantly Frequent schema churn breaks few-shot examples and cached schema context, so the model hallucinates removed columns; you need automated schema sync before text to SQL is reliable.
Quick scan · 2–4 min
Everything important, quickly.
Open with the mismatch: relational questions are exact and aggregative, while vector similarity returns approximately similar rows, so nearest-neighbor cannot sum, filter, join, or count. Introduce text to SQL as the right pattern — generate SQL, execute it, return the precise result — and stress that the exact computation belongs in the query engine. Reframe retrieval as over schema and exemplars, not data, and note that schema selection on a big database is itself a retrieval problem solved with embeddings. Cover correctness controls (grounding, validation, error-repair loop) and safety controls (read-only, timeouts, row limits, injection guards). Close on execution accuracy as the evaluation metric.
Real-world examples
Real products, models, and research that use this idea.
1Vanna and similar tools embed schema and curated example queries, then generate SQL grounded in that retrieved context
2The Spider and BIRD benchmarks score text to SQL by execution accuracy on the returned rows, not by matching the SQL string
3Analytics copilots in BI tools translate plain-English questions to SQL against a warehouse and run it under a read-only role
4Production stacks add a statement timeout and row limit so a model-written query cannot trigger a full-table scan that takes down the database
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYour database has 400 tables. How do you keep the schema in the model's context manageable?
Sign in or go Pro to see the approach for this follow-up.
QHow do you evaluate a text to SQL system, and why not just compare the generated SQL to a gold query?
Sign in or go Pro to see the approach for this follow-up.
QWhat is your defense if the model generates a destructive or runaway query?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Embedding every row and doing nearest-neighbor search for a question like 'total revenue in Q3' — similarity returns rows that look like the question, not the exact sum, which the database could compute precisely.