SQL Agents Are Finally Being Judged Like Systems, Not Chatbots With SELECT Privileges
The interesting thing about SQL agents is not that they can write a cleaner SELECT. We solved the “generate plausible SQL from a natural-language question” demo years ago, then spent the next few years discovering that plausible SQL is not the same thing as data work. Real data work is execution, repair, dialect handling, schema inference, metric clarification, and not confidently returning a number that came from the wrong join.
That is why C3 AI’s new DIA paper is worth paying attention to. The paper frames enterprise data interaction as a job for autonomous coding agents, not a specialized text-to-SQL component. DIA has three agents — a Data Interpreter, a Schema Creator, and a Query Generator — but the important architectural choice is that they behave like a system operating over a shared workspace and shared memory. The agent generates artifacts, executes them, inspects failures, and repairs its work. In other words: closer to how a competent data engineer behaves, and much less like a chatbot that got database credentials by accident.
The benchmark results are strong enough to make the pattern hard to ignore. DIA’s Query Generator was evaluated on 4,187 instances across seven SQL benchmarks: BIRD-Dev, BIRD-Critic, LiveSQLBench, BIRD-Interact, Spider2-Lite, Spider2-Snow, and Spider2-DBT. It spans SQLite, PostgreSQL, Snowflake, and DuckDB, and covers SQL generation, debugging, conversational interaction, and dbt project completion. That breadth matters because single-query text-to-SQL benchmarks have become too easy to overfit, too narrow to trust, or both.
The leaderboard jump shows up where the task stops being tidy
The headline numbers are not subtle. DIA reports 64.2% on BIRD-Critic versus a listed best baseline of 48.8%, 50.7% on LiveSQLBench versus 38.0%, 55.7% on BIRD-Interact versus 22.7%, 71.3% on Spider2-Lite versus 55.2%, 69.5% on Spider2-Snow versus 63.8%, and 37.5% on Spider2-DBT versus 35.3%. On BIRD-Dev, it lands at 77.7%, essentially tied with MARS-SQL at 77.8%.
The shape of those gains is more important than the average. The largest margins appear where the benchmark starts to resemble real work: +33.0 points on BIRD-Interact, +16.1 on Spider2-Lite, +15.4 on BIRD-Critic, and +12.7 on LiveSQLBench. That is exactly where a static SQL generator should struggle. Debugging, follow-up questions, multi-dialect execution, and project completion require state, not just syntax.
The system backbone is also worth noting. DIA uses OpenHands powered by Claude Sonnet 4.5, with o3 serving as the user simulator in BIRD-Interact. That means the result is not a clean measurement of “a SQL model” in isolation. It is a measurement of a harness: model, workspace, tools, memory, execution loop, and repair behavior. That is not a criticism. It is the point. Enterprise data agents will be products made of systems, not model weights plus a prompt template.
Text-to-SQL was always the wrong abstraction
Most production SQL generation failures do not happen because the model forgot how GROUP BY works. They happen because the business metric is underspecified, the schema naming is hostile, the column looks obvious but is stale, the warehouse dialect has one weird edge case, or the user asks a follow-up that changes the grain of analysis. DIA’s own weakness points in the same direction: the paper says the system is weakest on high-level questions that name a composite metric without defining the formula — 41.6% on LiveSQLBench and 47.6% on BIRD-Interact.
That is not a model trivia failure. That is a requirements failure. When someone asks for “active customers,” “net revenue retention,” “healthy pipeline,” or “qualified usage,” the correct answer is often not a query. The correct answer is a clarifying question, a metric lookup, a lineage check, or a refusal to invent the definition. If your data agent cannot represent that uncertainty, it will produce polished garbage faster than your analysts can review it.
The promising part of DIA is that it moves evaluation closer to the actual work product. A useful data agent should be able to inspect raw sources, create or validate schema assumptions, run queries, check result shapes, repair errors, preserve context across turns, and leave behind artifacts someone can review. SQL text is just one intermediate artifact. Treating it as the final product is like judging a CI system by whether it can write a shell command, not whether the build passes.
For engineering teams, the action item is straightforward: evaluate data agents end-to-end. Do not accept a vendor demo that shows one clean natural-language question becoming one clean query. Build an internal eval set with your warehouse dialects, your dbt models, your worst metric names, your permission boundaries, your incremental tables, and your favorite historical footguns. Score not only final correctness, but also whether the agent ran the query, handled errors, asked for clarification when needed, and produced inspectable artifacts.
The agent is only as trustworthy as its artifact trail
The paper’s direction is right, but teams should keep their procurement helmets on. The benchmark uses a strong proprietary model, and the authors note that granular per-instance results will be released after review. Until those details are public, the right stance is “promising architecture, rerun locally,” not “replace the analytics team by Friday.”
There is also a governance problem hiding inside the success. An autonomous data agent with workspace access can be much more useful than a text-to-SQL bot, but it also has more ways to fail. It can create derived artifacts, mutate project files, cache bad assumptions in memory, or generate queries that are technically valid but policy-invalid. The answer is not to avoid agents. It is to wrap them in the same controls we already know how to apply to data systems: scoped credentials, read-only defaults, reviewable diffs, lineage logs, query cost limits, and explicit approval for writes or exports.
The long-term shift is clear. The easy text-to-SQL demo will be commoditized. The durable value will come from execution-aware systems that understand artifacts, validation, dialects, user intent, and ambiguity. If your agent cannot run and validate the thing it emits, it is not doing data work. It is writing fan fiction about data work.
Sources: arXiv, OpenHands, BIRD benchmark, Spider2 benchmark