The order-intake agent went live in the spring. It reads purchase orders arriving by email, pulls out the customer, lines and delivery date, checks them against the customer master, and creates the order in the ERP. The monthly report on it looks healthy: most runs complete, model spend is inside budget, and the team that used to key orders by hand has moved on to other work.
Then someone mines a month of its runs, and three things come out that the report never showed. One run in five gets stuck in the same loop, calling the customer lookup again and again before handing the order to a person anyway. That loop accounts for about two-thirds of the agent’s model spend. And the orders it eventually creates still travel through the same slow credit check as before, so the customer waits just as long as they did when the order was keyed by hand.
None of this is an argument against agents. It is an argument for looking at them the way we look at any other process, because that is what they are.
An agent run is a case
An agent does not answer a question and stop. It works through a task in steps: it reads, it calls a model, it calls a tool, it decides, it calls another tool, it hands work to another agent or to a person. Some runs take the designed route. Some loop. Some wait. Some take a short cut nobody intended.
That is a process, and every run of it is a case. The three fields process mining needs, case, activity and timestamp, are already being written. Most agent frameworks now emit traces in the open tracing standard, OpenTelemetry, which records each agent run, each model call and each tool call as a span with a start, an end, and the tokens it consumed. A trace is an event log with different column names.
So the same discipline applies. Discovery reconstructs the routes agents actually take. Conformance checks them against the route they were designed to take. Performance analysis puts time and cost on every step. Nothing about this requires a new kind of tool; it requires treating agent traces as process data.
The one field most deployments are missing
A trace on its own tells you how the agent behaved. It does not tell you whether the business got what it needed. For that, the agent’s run has to be joined to the business case it worked on: the order, the invoice, the claim, the ticket.
Most deployments do not record that join. The trace has a run ID; the ERP has an order number; nothing connects them. Adding the business object’s ID to every trace is the single most valuable change a team can make before mining agents, and it is a small one.
Once it is there, two things become possible. The agent’s steps can be laid inside the business process, so that an intake run is seen as the first part of the order’s journey rather than a separate story. And the relationship can be many-to-many, as it really is: one email can create three orders, and one order can be touched by the intake agent, the credit agent and a service agent answering the customer’s query. Modelling cases around several objects at once, rather than forcing everything into one case ID, is what keeps that picture honest.
What the map shows
Mine a month of intake runs, joined to the orders they created, and the map shows what the run report averaged away.
Variants. About 72 per cent of runs take the designed path: read, extract, look up customer, validate, create order. About 9 per cent hand the order to a person straight away, correctly, because the document is unreadable or the customer is new. The remaining 19 per cent are the problem.
Loops. In that 19 per cent, the customer lookup is called a median four times. The cause, once the tool calls are inspected, is mundane: a group of long-standing customers still send purchase orders with an old-format customer number, the lookup returns nothing, and the agent tries variations until it gives up. Over 80 per cent of these runs end with a person anyway.
Hand-offs. Which agent passed work to which, how often, and how long the work waited in between. Hand-offs between agents behave like hand-offs between teams: they are where time disappears.
Escalations. When, why and after how many steps a run is handed to a person. An escalation on the first step, for a good reason, is the system working. An escalation after the fifth retry is a cost that bought nothing.
Deviations. Runs that skipped the validation step, created an order before the delivery date was confirmed, or took an action outside the agent’s stated scope. These are conformance findings, checked against the designed route exactly as they would be for a person doing the same job.
What an agent really costs
Agent cost is usually reported as a monthly total: tokens consumed, multiplied by price. That number is real, but it answers the wrong question. The question that matters is what each successful outcome costs, with everything that went into it.
That has four parts:
- Model calls, the tokens in and out of every call in the run.
- Tool calls, where the tools are priced, rate-limited or slow.
- Human time, every minute a person spends picking up an escalation, checking a result or correcting it.
- Rework downstream, when an agent’s output causes a later step to be repeated.
Mined by variant, the picture is stark. A designed-path intake run costs a few cents. A loop run costs around eleven times that in model calls alone, then several minutes of a person’s time on top, and still ends in a manual order. Because the loop runs are counted against the outcome they failed to deliver, cost per successful order is far higher than the monthly total suggests, and almost all of the difference sits in one variant with one cause.
This is the same pattern the dashboard misses in ordinary processes: a healthy average hiding a minority of cases that carry most of the cost. Agents do not change the pattern. They make it more expensive.
From finding to fix
Because every finding is attached to cases, steps and causes, the improvements it points to are specific. These are the ones that come up most often.
Fix the tool, not the prompt. The intake loop is not a reasoning problem, and rewriting the prompt will not solve it. Normalising old-format customer numbers inside the lookup tool removes the loop, most of the escalations and about two-thirds of the spend in one change.
Route by variant. Most purchase orders are clean and predictable. The map shows which variants a smaller model, or a plain rule, handles as reliably as the large one. Send those there, and keep the expensive model for the variants that need it.
Cap and hand over. The log shows that runs which have not succeeded by the second retry almost never succeed at all. Stop at two, and hand the order to a person with the context already assembled: what was tried, what failed, and why.
Move the approval to where the deviation is. If one action is approved by a person 99 per cent of the time, that approval is slowing every case to catch almost nothing. If another step is where agents deviate, that is where the human check belongs. Conformance data says which is which.
Fix the process before the agent. The orders the intake agent creates still loop through the credit check, because the credit limit for one channel is refreshed weekly. Automating the step in front of a bottleneck makes orders arrive at the bottleneck faster. The agent inherited the process’s problem; mining the two together is what shows it.
Execution intelligence: acting while the run is open
Everything above is analysis of runs that have finished. Execution intelligence applies the same models to runs that are still going.
An agent run on its third lookup retry, for an order above the value threshold, from a customer whose orders loop more often than most, can be scored as it happens. If it is very likely to fail, it is stopped and handed to a person now, rather than after another three calls. A run whose cost has passed what a successful run of its variant normally costs triggers an alert on that case, not at the end of the month. And the action taken, whether by the agent, by a rule or by a person, is written back into the same log, so the next model learns from what actually happened.
This is also where agents and people stop being two separate systems to supervise. The same live process model shows which open cases are at risk, whoever is working them, and the same conformance checks apply to both.
Context: what the agent should read before it acts
The deeper point is that the process model is not only a way to watch agents. It is the context they need to do the work well.
An agent given the procedure document knows how the process is supposed to run. An agent given the live process model knows how it is running:
- Where this case is, what has already happened to it, and what has already been tried.
- What normally happens next for cases like this one, and how long it usually takes.
- What is already late, or at risk, and which action has worked for similar cases before.
- How agents perform on this variant, so an agent can tell when it is on a route where it usually fails, and hand over early.
The intake agent, with that context, would know that this customer’s orders often fail the lookup and that the last two hundred were resolved by mapping the old number. The credit agent would know the limit it is about to check against is stale for this channel. Neither needs to be a better model to do that. They need to read better data.
Where to start
Start with the runs you already have. Three things are needed:
- Traces with tokens, which most agent frameworks already produce.
- The business object’s ID on every trace, which most do not, and which is a small change.
- The business events, from the ERP or workflow system the agent writes into, so the agent’s steps sit inside the process they serve.
With those in place, a month of runs is enough to show where agents loop, where they deviate, and what each successful outcome really costs. Analysts expect a large share of agentic AI projects to be cancelled over rising costs and unclear value. Measuring cost and value per case, from the first month of running, is the most direct way not to be one of them.
If the agents sit on a process you have not mined yet, start there. The readiness check takes five questions. If the event history is there, a two-week pilot maps the process from your real data, and shows where agents will help, and where they would only make a loop run faster.