GitHub’s Qubot Shows Enterprise Agents Need Curated Context, Evals, and Boring SQL

The most useful sentence in GitHub’s Qubot write-up is not the one about plain-language analytics. It is the part where the company admits that structured, curated context made the internal agent more accurate and three times faster at returning the right answer.

That is the enterprise-agent story hiding under the chatbot story. GitHub did not build a magic “ask your data anything” box and declare victory. It built a context system, an evaluation loop, a contribution model, and MCP boundaries around Kusto and Trino. The natural-language interface is the part users see. The boring machinery is the part that makes the answer survivable.

Qubot is GitHub’s internal Copilot-powered analytics agent. Employees can ask questions about GitHub’s data warehouse in plain English and get answers in seconds. GitHub says the tool now has hundreds of enthusiastic users running thousands of queries, and that it has dramatically reduced questions in analytics Slack channels. That is a nice adoption story. The more important story is how much discipline it took to make “chat with data” not collapse into demo slop.

The agent is not the architecture

Qubot is available through Slack, VS Code, and Copilot CLI. The Slack path is the clever default: a user posts a question in the Qubot channel, a Qubot instance starts as a Copilot Cloud Agent running on github.com, and the answer comes back in Slack. Results are also stored as Markdown reports in pull requests, so the answer does not disappear into chat history. That turns an ephemeral response into an artifact someone can review, refine, and reuse in a dashboard.

That artifact choice matters. Analytics answers are claims about the business. They need provenance. If someone asks which cohort retained best or which product moved a metric last week, the answer should not be a floating paragraph from a model. It should be tied to query logic, context, assumptions, and a reviewable trail. Markdown in a pull request is not glamorous, but it is much closer to how engineering organizations should treat agent outputs that influence decisions.

The context layer is where GitHub’s design gets serious. The warehouse is organized around bronze, silver, and gold data. Bronze means raw events, with telemetry context contributed by product teams. Silver means conformed facts and dimensions, with query examples, usage guidance, and mandatory filters maintained by data and analytics. Gold means curated business datasets, with metric definitions and business rules contributed by dataset owners.

That split is the opposite of “dump the schema into the prompt.” It recognizes that different layers of data require different owners and different kinds of knowledge. Raw event metadata is not the same as a business definition. A mandatory filter is not the same as a dashboard metric. If those distinctions are blurred, the model may still write syntactically valid SQL while answering the wrong question.

Context engineering becomes data engineering

GitHub says the context layer is loaded at runtime through the GitHub MCP Server. It also uses a separate context agent to ingest, organize, and normalize Markdown knowledge from repositories into a structured format that evaluations have shown to work for Qubot. This is the part most enterprise-agent projects skip because it does not fit neatly on a launch slide.

For practitioners, the lesson is blunt: if nobody owns the context, nobody owns the answer. GitHub’s hub-and-spoke model makes product teams own telemetry knowledge, analytics teams own silver-layer guidance, and business or data owners own gold-layer definitions. Contributions can flow through repositories and pull requests. That gives the organization a way to update the agent without treating prompt edits as folklore.

The “three times faster” result is also more than a latency brag. Better context reduces search space. It helps the agent pick the right dataset, grain, filter, and query engine sooner. That means fewer failed tool calls, fewer tokens, less waiting, and fewer opportunities for the user to accept a confident wrong answer. Context quality is cost control. It is also reliability engineering.

GitHub’s evaluation framework is the other piece worth copying. Every context or configuration change is evaluated before shipping. The benchmark uses curated prompts with known correct answers and ground-truth SQL; automated orchestration via gh agent-task create; multiple parallel trials; JSON result storage; and aggregation for completion rate, accuracy, and duration. That is not exotic. It is exactly the minimum viable discipline for a stochastic system that answers business questions.

Teams building data agents should steal this pattern before they write their first internal announcement. Define common questions. Store ground-truth SQL. Run multiple trials because agents are not deterministic. Track regressions when context changes. Measure latency and completion, not just correctness. If a context update makes one domain better and another worse, you want to know before the CFO asks the agent for a number.

MCP is a policy boundary, not a cable

Qubot queries Kusto and Trino through MCP. GitHub built a custom Trino MCP server and deployed a local version of Microsoft Fabric RTI’s MCP Server for Kusto. The routing logic defaults to Kusto and switches to Trino when the question requires more complex joins or historical analysis.

That is a sensible architecture, but it also puts MCP where it belongs: directly on the trust boundary. An agent that can query analytics systems is not merely “integrated.” It has access to sensitive operational and business data under a user’s permissions. Microsoft’s Fabric MCP documentation is unusually direct about this class of risk: MCP clients can invoke operations based on user RBAC, autonomous or misconfigured clients may perform destructive actions, and compliance responsibility extends beyond the data platform when clients and model providers enter the loop.

For Qubot, the most relevant mitigation is probably scope. Analytics query access should be read-focused, logged, and wrapped in least-privilege permissions. The agent should not need broad warehouse powers to answer exploratory questions. If a workflow eventually supports write actions, dataset mutations, dashboard changes, or scheduled jobs, those should require a different approval path. “The model asked nicely” is not an authorization model.

The broader point is that enterprise data agents do not fail only by hallucinating. They fail by querying the wrong layer, using the wrong metric definition, ignoring a mandatory filter, leaking sensitive data into the wrong surface, or making an answer too hard to audit. Qubot’s architecture addresses those failure modes with context ownership, PR-backed artifacts, evals, and MCP boundaries. That is why the write-up is more valuable than another vendor demo.

If you are building something similar, start smaller than your ambition. Pick one domain. Create bronze/silver/gold-style ownership, even if your names differ. Put context in repositories. Require review. Build ten painful test questions with known answers. Track accuracy, completion, and duration. Only then put the agent where people already ask questions, whether that is Slack, Teams, a CLI, or the IDE.

Qubot’s real lesson is not that every company needs an analytics chatbot. It is that agents get useful when the organization turns its implicit knowledge into maintained infrastructure. The input box is the easy part. The hard part is making sure the answer deserves to exist.

Sources: GitHub Blog, GitHub MCP Server, GitHub CLI manual, Microsoft Fabric RTI MCP overview