Artificial intelligence is crossing an important line. It is moving from technology that tells people what they could do to technology that can increasingly do it for them.
That distinction could become one of the biggest governance challenges facing businesses as they move from generative AI and copilots to autonomous AI agents capable of accessing systems, initiating transactions, communicating with customers, changing data and executing business processes. The issue moved sharply into focus this week when Nvidia launched a new AI safety platform designed specifically to prevent AI agents from operating outside the boundaries organisations have set for them.
According to Dr Pierre Le Roux, Managing Director of digital consultancy MOYO, the development reflects a much bigger shift taking place inside enterprises.
“Until now, much of the AI conversation has been about the accuracy of the answer,” says Le Roux. “Agentic AI changes the question. If the technology is able to take an action, the issue is no longer simply whether the answer was right or wrong. You have to ask who gave it permission to act, what it was allowed to access, where its authority ended and ultimately who is accountable for the outcome.”
This is becoming urgent as businesses move beyond AI experimentation. Gartner has predicted that by 2027, 40% of enterprises could demote or decommission autonomous AI agents because governance gaps are only discovered after the systems have been deployed. Le Roux says organisations should take that warning seriously.
An AI agent is not just another piece of software
Traditional software generally operates according to predefined rules. An AI agent can interpret a goal, decide what steps are required and use different systems or tools to complete those steps. That is what makes agents potentially transformative. It is also what makes them fundamentally different from the technology businesses have governed in the past. Consider an AI agent tasked with resolving customer queries.
At a relatively low level of autonomy, it might retrieve information and recommend a response to an employee. At a higher level, it could communicate directly with the customer, access their account, approve a refund, change an order and update multiple systems without human intervention. “The business value increases dramatically as you give the agent more authority, but so does the consequence of getting something wrong,” says Le Roux. “The mistake is to think about this purely as an AI problem. It is an operating-model problem. You are effectively delegating authority to a digital actor.”
Give AI authority, not unlimited access
Le Roux says businesses will increasingly need to define decision rights for AI in much the same way that they define them for people. An employee does not generally receive unrestricted access to every system, customer record and financial process in an organisation simply because they may need information to do their job. AI agents should be no different. Each agent should have a clearly defined role, access only to the data and systems required for that role, limits on the decisions it can make and clear rules determining when an action must be escalated to a person.
More importantly, organisations need to be able to trace what happened afterwards. “If an agent makes 10,000 decisions overnight, saying that a human was technically responsible for supervising it is meaningless unless the organisation can see what the agent did, why it acted, what information it accessed and whether it operated within its authority,” says Le Roux.
Governance therefore cannot simply be a policy document written before deployment. It needs to be engineered into the systems themselves through identity management, permissions, audit trails, monitoring, escalation mechanisms and the ability to stop or reverse actions when something moves outside agreed boundaries.
Humans remain accountable
That does not mean every action an AI agent takes should require human approval. Doing so would remove much of the productivity benefit businesses hope to gain from autonomous systems. Instead, Le Roux argues that organisations need to determine where human judgement genuinely matters.
An AI system automatically scheduling a routine maintenance inspection poses a very different level of risk from one approving a multimillion-rand payment, changing an employee’s remuneration or making a decision affecting a customer’s access to a financial product. “The more consequential the decision, the stronger the governance needs to become,” he says. What businesses cannot do is outsource accountability to the technology.
“AI may execute the decision, but accountability remains with the organisation that designed the process, gave the agent access and determined its authority.” That is why Le Roux believes the conversation around AI governance needs to change quickly. Businesses are no longer governing systems that merely produce information for humans to consider.
They are beginning to govern systems that can act.
“The question is no longer whether you trust AI,” he says. “The question is whether your organisation has engineered the boundaries within which AI is allowed to act.”
