The sales director types a question into the new assistant in the BI tool: what was revenue in the Gulf last quarter? Four seconds later it answers, $48.2 million, with a tidy chart. The finance pack, published the same morning, says $44.9 million.
Neither number is invented. The assistant added up booked orders, including one that was later cancelled, at today’s exchange rate. Finance reports invoiced revenue, net of returns, at the month-end rate, and excludes a sale to a sister company. Ask the assistant again with slightly different words and it returns a third figure. Nobody in the meeting can say which rules any of them followed.
That is the real risk in letting AI answer questions about your numbers. Not that it is wrong now and then, which people can live with, but that nobody can tell when.
Why the model has to guess
When an agent answers a business question from a database, it writes a query. To write it, the model sees table names, column names and perhaps a few sample rows. What it does not see is the meaning that the business has attached to them over the years.
“Revenue” alone hides at least five decisions. Booked or invoiced. Gross or net of returns and discounts. Which date decides the quarter, and whose fiscal calendar. Which exchange rate. Whether sales between group companies count. “The Gulf” is another: which countries, and is the customer’s region where it is headquartered or where the goods were delivered?
Each of those was decided long ago, by someone in finance or sales operations, and written into a report definition, a spreadsheet or a stored procedure. A person who has worked there for a year knows them. A model meets them fresh every time, infers them from column names, and infers them differently depending on how the question is worded. The result is a fluent answer whose rules change from one question to the next.
What the evidence shows
Research on answering business questions from company databases keeps finding the same thing. The model is rarely the limit. The context is.
- Raw tables defeat even strong models. In a benchmark built on an insurance data model, GPT-4 answered 16.7 per cent of 43 business questions correctly when it queried the database tables directly. When the same data was described through a model of the business’s concepts and how they relate, accuracy rose to 54.2 per cent. On the raw tables it got none right once a question needed more than four tables.
- Checking against the definitions helps again, including knowing when to stop. A follow-up study checked every generated query against that business model with fixed rules, and asked the model to repair any query that broke them. Accuracy rose to about 72 per cent, and in a further 8 per cent of cases the system said “I don’t know” instead of guessing.
- The best scores come where the meaning is supplied. On the public BIRD benchmark, the best system scores about 83 per cent against a human baseline of about 93 per cent, as of November 2026, and the leading entries are given hand-written hints explaining what the business terms mean. On Spider 2.0, built from real enterprise tasks on databases that often have more than a thousand columns, the strongest agent solved 21.3 per cent of tasks when the benchmark was published. The near-perfect scores since then are on the version that supplies curated metadata and documentation.
- The analysts have reached the same conclusion. Gartner predicts that by 2027, organisations that prioritise semantics in AI-ready data will increase the accuracy of their AI agents by up to 80 per cent and reduce costs by up to 60 per cent.
The pattern is consistent. Models get the numbers right when someone has written down what the numbers mean, and when the answer is checked against it. That is the job of a semantic layer.
What a semantic layer gives an agent
A semantic layer is one governed place where the business meaning of the data is written down once: each metric and its formula, each dimension and its hierarchy, how the tables join, the fiscal calendar, the currency rules, and who may see which rows. Dashboards, spreadsheets and models read from it, so they all get the same answer.
Put an agent in front of it and the mechanism changes. The agent no longer writes free-form queries against raw tables. It translates the question into a request the layer understands, a metric, some dimensions, a filter and a period, and the layer generates the query from definitions that finance has already agreed. The agent’s job shrinks from “write any query that looks plausible” to “pick the right defined metric”, which is a far smaller space in which to make mistakes.
Four things follow.
- The same number as finance. Revenue means net invoiced revenue because that is the only revenue the layer offers. The agent cannot drift into booked orders on its own.
- Answers that show their working. Every answer can carry the metric used, the filters, the period and how fresh the data is. A person can check it in seconds, and an auditor can check it later.
- Access rules that travel. The row-level security that stops a regional manager seeing another region’s margin applies to the agent exactly as it does to the dashboard, because both go through the same layer.
- An honest “I don’t know”. When a question needs a metric nobody has defined, the agent can say so, instead of improvising one. That gap goes on the backlog rather than into a board paper.
Five checks before you switch it on
Most organisations now have an assistant that could be pointed at their data tomorrow. Before it is, five checks separate a useful tool from a source of arguments.
- Collect the real questions. Ask the people who will use it for the twenty or thirty questions they actually ask each month, and ask finance for the answer each one should get. That list is your test set.
- Define the metrics behind them, once. Each metric gets one definition, one owner and one place it lives. If two departments disagree about a definition, settle it before the agent inherits the argument.
- Make it show its working. Every answer should state the metric, filters, period and data freshness. If the tool cannot do this, people will check its answers by hand, which removes the point of having it.
- Apply the same access rules. The agent should see exactly what the person asking would see in a dashboard, enforced in the layer and not in the prompt.
- Test it like a release. Run the test set every time a definition, model or data source changes, and keep a log of the questions it could not answer. That log tells you what to define next.
The work is mostly data engineering
A semantic layer is only as good as the tables underneath it. It needs one customer hierarchy instead of five spreadsheets, a fiscal calendar table, agreed exchange rates, returns and cancellations that arrive in time, and dimensions that mean the same thing in every subject area. That is data estate work, and most of the effort of a reliable agent sits there, not in the prompt.
The good news is that it is done once. The same definitions serve the dashboards, the planning model and every agent that follows, which is why we argued in where the AI budget is going that a shared semantic layer is one of the few AI investments every later use case inherits. And when the next question is not “what was revenue” but “why are orders late”, the definitions need to be joined by live process context, which shows where each case is and what is holding it up.
Where to start
Do not start with the whole business. Pick one domain where questions are frequent and the numbers matter, usually sales or finance. Take its twenty questions, define the metrics behind them in the semantic layer, connect the agent to the layer and not to the raw tables, and run the test set until it gives finance’s answer every time. Then let people use it, read the log of what it could not answer, and widen from there.
An agent built that way answers fewer questions in its first month than one pointed straight at the warehouse. Every answer it does give is one the business can use. That is how business intelligence earns trust, with or without AI.