Table of Contents
Text-to-SQL lets a language model turn a plain-language question into a SQL query, run it and return the result. On a production database it is workable when the model queries curated, read-only views on a replica, under its own restricted account, with the SQL shown to the user and tested against known answers beforehand.
Key takeaways
- Benchmark scores overstate real accuracy. An agent built on o1-preview solved only 21.3% of the enterprise tasks in Spider 2.0; its authors compare this with 91.2% on the original Spider benchmark (Lei et al., 2024).
- Production schemas are the hard part: hundreds of tables, abbreviated column names, ambiguous business terms and codes only experts understand (Floratou et al., 2024).
- Treat generated SQL as untrusted input. OWASP lists unchecked LLM-generated SQL as an example of improper output handling and advises treating the model like any other user (OWASP LLM05:2025).
- Read-only access blocks write attacks but not everything: instructions stored in database text fields can still mislead the model (Pedro et al., 2025).
- Show the SQL with every answer, and test on real questions with known answers before anyone relies on the figures.
What is text-to-SQL?
A text-to-SQL system takes a question such as "How many corrective work orders did we open on the feed pumps last quarter?", gives a language model the question plus a description of the relevant tables, receives a SQL query, runs it and returns the result, often as a table or chart.
In a plant, the useful databases are the maintenance system, the LIMS, production and shift reporting, inventory and purchasing. Most staff cannot write SQL against them, and the questions they ask are rarely covered by an existing report.
In the vocabulary of our article on chatbots, copilots and AI agents, a text-to-SQL assistant is a chatbot with one tool: it reads data and changes nothing. Keeping it that way is most of the safety work.
How accurate is text-to-SQL today?
Accuracy is usually measured as execution accuracy: the generated query is run and its result is compared with the result of a reference query. Three public benchmarks show how much the answer depends on the database.
| Benchmark | What it contains | Reported result |
|---|---|---|
| Spider (2018) | 10,181 questions, 5,693 SQL queries, 200 databases in 138 domains (Yu et al.) | 91.2%, the figure the Spider 2.0 authors cite for comparison (Lei et al.) |
| BIRD (2023) | 12,751 question-SQL pairs, 95 databases, 33.4 GB, 37 professional domains (Li et al.) | 40.08% for ChatGPT at publication, against 92.96% for humans; best leaderboard entry 83.40% on the test set (BIRD leaderboard, checked November 2026) |
| Spider 2.0 (2024) | 632 enterprise workflow tasks, on databases often containing over 1,000 columns, in systems such as BigQuery and Snowflake (Lei et al.) | 21.3% for an agent built on o1-preview, at publication |
Two cautions. The leading BIRD entries are marked as using "oracle knowledge", the hints the benchmark supplies to link questions to database values; your users will not provide such hints. And these are public databases with clean documentation. Scores on your own schema can be lower, and only your own test set will tell you.
Why are production databases harder than benchmarks?
Schema size and naming. Floratou and colleagues, from Microsoft and a text-to-SQL startup, describe one internal financial data warehouse with 632 tables, more than 4,000 columns and 200 views, with abbreviated names and custom terminology "that only domain experts can understand" (Floratou et al., 2024). ERP and maintenance systems look similar. The same paper found that giving the model only the relevant tables and columns improved accuracy, even when the whole schema fitted in the prompt.
Ambiguity. When the authors reviewed the 272 questions of the KaggleDBQA benchmark, 112 of them, 41.1%, were ambiguous, and the two human annotators agreed on which ones only about 62% of the time. In a plant, "production last month" can mean gross or net, tonnes or cubic metres, calendar or fiscal month, one line or the whole site.
Questions the database cannot answer. Users do not know what each column means. In the paper's example, a year column holds a singer's first album year, but a model filters on it to answer a question about the year of death. In the same study, given questions the database could not answer, GPT-3.5 still wrote a query for 40% of them, typically with invented table and column names; GPT-4 recognised all of them.
Dirty values and hidden knowledge. The BIRD authors point to dirty database contents, the external knowledge needed to link questions to values, and query efficiency on large databases (Li et al., 2023). Plant data has all three: status codes, equipment tags written three ways, units that change between tables.
What does a safe setup look like?
The LangChain SQL agent guide states the problem plainly: answering questions over a database "requires executing model-generated SQL queries", narrow permissions "will mitigate, though not eliminate" the risks, and it "can be prudent" to check queries before they run. A practical setup has seven layers.
- Query a copy, not the primary. Run the assistant against a read replica or reporting database. SQL Server, for example, can offload "reporting and analytics workloads to read-only replicas" to free resources on the primary (Microsoft Learn). A bad query then cannot slow down the system that planners and operators use.
- Give it its own account, with access only to curated views. OWASP's example for a database extension is that it "should not have access to other tables, nor the ability to insert, update or delete records", enforced through database permissions (OWASP LLM06:2025). Broad roles are too much: PostgreSQL's
pg_read_all_datareads all tables, views and sequences (PostgreSQL docs). - Filter rows by user. If users have different rights, the database should enforce them. PostgreSQL row security policies restrict "on a per-user basis, which rows can be returned"; superusers and roles with
BYPASSRLSbypass them, and table owners normally do too (PostgreSQL docs). - Validate before executing. Parse the query, accept a single
SELECTon allowed views only, and add a row limit. OWASP's advice is to treat the model "as any other user, adopting a zero-trust approach" (OWASP LLM05:2025). - Set a timeout on the assistant's role. PostgreSQL's
statement_timeoutaborts any statement that runs longer than the limit; the documentation advises against setting it inpostgresql.confbecause it would affect all sessions (PostgreSQL docs). Note thatdefault_transaction_read_onlyonly sets a default; permissions are the control. - Show the SQL and the views used with every answer. The person asking, or a data owner, can then spot a wrong filter. Anything beyond reading, such as exporting or sending results, waits for approval, as described in human-in-the-loop for AI agents.
- Log the question, the SQL, the user and the row count. Our guide to securing an on-premise LLM lists what else to record and how to protect the log.
If the views contain personal data, such as technician names or employee numbers, Algeria's Law 18-07 applies. According to the CMS guide, Article 38 requires security measures commensurate with the risks, and Law 25-11 of 24 July 2025 adds an automated logbook of processing operations and breach notification within five days. Confirm with legal counsel how this applies to your system.
Does read-only access stop prompt injection?
Partly. Pedro and colleagues tested what they call prompt-to-SQL injection on applications built with LangChain and found the attacks worked across the seven language models they evaluated (Pedro et al., 2025). Restricting the assistant to SELECT blocked the write attacks. But table-level permissions "can result in coarse protection granularity", leaving sensitive records in permitted tables exposed.
The paper also shows indirect attacks. An external user writes an instruction into a job posting description; when someone later asks about job postings, the model reads that text and gives a false answer. In a plant, the equivalent fields are work order comments, supplier names, lab remarks and anything typed by contractors.
The responses follow from that:
| Risk | Example | Control |
|---|---|---|
| Write or delete through generated SQL | A user asks the assistant to "clean up" old records | SELECT-only account on views; query validation |
| Reading rows the user may not see | A planner asks about salaries joined to work orders | Views without sensitive columns; row-level security |
| Instructions hidden in data | A comment field says "ignore previous instructions" | Exclude free-text columns the question does not need; treat returned text as data |
| Expensive query on a live system | A join across years of history at shift change | Replica; statement timeout; row limit |
| Plausible but wrong figure | "Production" summed across units | SQL shown with the answer; tested question set |
The wider defences against prompt injection are covered in securing an on-premise LLM.
How do you give the model business context?
A raw schema rarely carries the meaning. Snowflake's documentation for its text-to-SQL service says generic AI solutions "often struggle with text-to-SQL conversions when given only a database schema", because schemas lack business definitions and metrics; its semantic views add descriptions, synonyms, metrics and predefined join paths (Snowflake docs).
You can do the same on your own servers:
- Views with readable names, one per business object: closed work orders, lab results per sample point, daily production per line.
- Column descriptions with units, and the meaning of every status code.
- A glossary of plant terms and their synonyms in the languages staff use. Our industrial AI glossary is a starting point for the AI side.
- Worked examples: a few dozen real questions with their correct SQL, written by someone who knows the data.
This curation is the bulk of the work, and it belongs to the data owners, not only to IT.
How do you test it before people rely on it?
Collect real questions from planners, lab staff and production engineers, and have a data owner write the reference SQL for each. Score by comparing results, not query text. Include questions that are ambiguous, where the right behaviour is to ask which unit or period is meant, and questions the database cannot answer, where the right behaviour is to say so. Re-run the set after every model update and every schema change. Our guide to evaluating an LLM on your own documents covers test set size and grading.
Text-to-SQL complements document search; it does not replace it. Questions about counts, totals and trends need SQL; questions about procedures need RAG over manuals.
What we have seen at KABAS AI
DALILE lets users ask questions in plain language and handles the SQL queries, connects to SQL databases, SAP, Oracle and LIMS, and runs on the client's own servers. Each new database connection raises the questions in this article, so we recommend three steps before any pilot:
- Start with one database and a short list of views agreed with its owner, rather than the whole schema.
- Write the test questions before the connection, with the people who will use it, including the ambiguous ones.
- Decide who checks the SQL for figures that feed a decision or a report.
Our agentic AI team scopes assistants and their tools, and our data science team can help prepare and document the data the views depend on.
Sources
- Lei et al., Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows, arXiv 2411.07763, 2024
- Yu et al., Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task, arXiv 1809.08887, 2018
- Li et al., Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (BIRD), arXiv 2305.03111, 2023
- BIRD-SQL leaderboard
- Floratou et al., NL2SQL is a solved problem... Not!, CIDR 2024
- Pedro et al., From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?, arXiv 2308.01990, ICSE 2025
- OWASP GenAI Security Project, LLM05:2025 Improper Output Handling
- OWASP GenAI Security Project, LLM06:2025 Excessive Agency
- LangChain documentation, Build a SQL agent
- Snowflake documentation, Cortex Analyst
- PostgreSQL documentation, Row Security Policies
- PostgreSQL documentation, Predefined Roles
- PostgreSQL documentation, Client Connection Defaults
- Microsoft Learn, Use read-scale with availability groups
- CMS Expert Guide to data protection and cyber security laws: Algeria
Frequently asked questions
Is it safe to connect an LLM to our production ERP or maintenance database?
Not directly. Point it at a read-only replica or reporting copy, give it its own database account that can only read a set of curated views, and set a query timeout. Treat every generated query as untrusted input, validate it before it runs, and log the question, the SQL and the user.
Does a read-only database account make text-to-SQL secure?
It blocks the worst case, an attacker making the model change or delete data. It does not stop the model from reading rows the user should not see, or from being misled by instructions hidden in free-text fields such as comments. Narrow views, row-level security and treating returned text as data are still needed.
What is the difference between text-to-SQL and RAG?
RAG retrieves passages from documents, such as manuals and procedures, and the model writes an answer from them. Text-to-SQL writes a query against structured tables, such as work orders, lab results or production figures, and the answer is the query result. Questions about counts, totals, trends and lists usually need SQL; questions about how to do something usually need documents.
Can text-to-SQL replace our dashboards and reports?
No. Dashboards and validated reports remain the reference for recurring figures such as monthly production or KPIs. Text-to-SQL is useful for the one-off questions nobody built a report for. When it gives a figure that matters, compare it with the official report or have a data owner check the query.
How do we know if the generated SQL is correct?
Show the SQL and the tables it used next to every answer, so a person who knows the data can check it. Before go-live, test the system on a set of real questions with known answers, including ambiguous questions and questions the database cannot answer, and repeat the test after every model or schema change.



