When AI agents make decisions, who determines whether those decisions are appropriate?
By Gerry Blass and Bill Lewis
Your AI agent may be making decisions you never explicitly authorised.
You may have approved what the agent is allowed to do, which systems it can access and which rules it must follow. But have you established how it should choose between competing courses of action? Who determines whether those choices are appropriate, and what evidence do you have that the agent makes them well?
If your organisation is deploying autonomous AI agents, these are governance questions you cannot afford to leave unanswered.
Imagine a hospital outpatient clinic that has to cancel an entire day’s appointments.
An AI agent is instructed to reschedule the patients. It checks availability across several clinics, reviews the waiting list, considers appointment priorities, offers alternatives and updates the hospital’s scheduling system.
Then it encounters a decision that is less straightforward.
Two patients need the same limited appointment slot. One has been waiting considerably longer. The other requires a time-sensitive follow-up. The agent can prioritise the longest wait, allocate the slot to the second patient, search for another alternative or refer the decision to a clinician.
Each option has a rationale. None is simply a matter of following a booking instruction.
Which consideration should take precedence? Does the agent have enough information to decide? Should it seek additional clinical information, or is this a decision it should not be making at all?
These are questions of judgement.
The hospital may have approved the agent, tested its accuracy, restricted its access to information and established procedures for monitoring its performance. Yet none of those safeguards necessarily establishes whether the choices it makes between competing priorities are appropriate.
Who decided what constitutes good judgement in these circumstances? Who tested the agent’s ability to exercise it? And who authorised it to make such decisions?
As AI moves beyond producing recommendations into performing tasks and making operational decisions, organisations need to govern more than what their systems are permitted to do.
They must also consider how those systems choose what to do.
Following the rules is not always enough
Governance has traditionally relied on policies, procedures, standards, authority limits and controls. These establish how an organisation expects work to be carried out and who is responsible for the consequences.
AI governance has added further requirements: evaluating systems, testing performance, protecting information, monitoring behaviour, managing risks and maintaining human oversight.
All of these are necessary.
But autonomous AI agents introduce an additional challenge.
An agent may be given an objective rather than a fixed series of instructions. It can retrieve information, decide which steps to take, use software tools, respond to changing circumstances and carry out actions across several systems.
In doing so, it may encounter situations where multiple actions are possible, information is incomplete or different objectives compete.
The agent must select a course of action.
We can call that decision-making, reasoning or optimisation. Whether it amounts to judgement in the human sense is a philosophical question. For governance purposes, the terminology matters less than the responsibility.
When an AI agent is authorised to choose between consequential courses of action, the organisation must establish how the appropriateness of those choices will be assessed.
It is not enough to know that the task was completed.
Three different things need to be tested
Consider the hospital scheduling agent.
The first question is whether it completed its assignment. Did it contact the correct patients, find suitable appointments, update the records and confirm the bookings?
These are measures of performance.
The second is whether it complied with the rules. Did it respect patient privacy, use only authorised systems, follow scheduling restrictions and obtain approval where required?
These are measures of compliance.
The third question is different.
When the agent encountered competing priorities, did it take account of the right considerations? Did it recognise uncertainty? Did it distinguish circumstances where it could reasonably decide from those requiring professional intervention?
These are questions about the appropriateness of its decisions.
The distinctions matter because an agent could perform well on the first two measures while making an unacceptable choice on the third.
An agent can achieve a 100% task-completion rate and still exercise unacceptable judgement.
If completion is the principal measure of success, you may be rewarding the agent for making decisions it should have escalated.
That is a governance risk that conventional measures of productivity and technical performance may fail to expose.
The hospital therefore needs to decide not only what the agent is allowed to do, but what it is competent to decide.
Judgement is becoming a governance issue
The need to examine AI decision-making is increasingly recognised.
The National Institute of Standards and Technology’s AI Risk Management Framework addresses human oversight, system performance, risk assessment and continuing monitoring.
The Joint Commission has introduced a voluntary Responsible Use of AI in Healthcare certification programme, recognising the importance of organisational governance, safeguards and continuing evaluation.
And in July 2026, the Coalition for Health AI published a dedicated testing and evaluation framework for agentic AI. It addresses matters including the correctness of intermediate actions, compliance with policies, safety, fairness and reliability.
These are significant developments. They demonstrate that the industry is establishing methods for evaluating what autonomous systems do, not merely what their final outputs look like.
But establishing methods for evaluating an agent is not the same as deciding whether its judgement is acceptable for a particular responsibility in a particular hospital.
That requires the organisation to translate general principles and technical assessments into the circumstances of its own operations.
An agent may be technically successful while making decisions against inappropriate objectives, incomplete criteria or priorities that nobody deliberately approved.
The question is whether the organisation has made the basis for those decisions explicit, and established how their acceptability will be tested.
Your technology supplier can demonstrate what an agent does. It cannot decide what constitutes acceptable judgement for your organisation.
That responsibility belongs to the people accountable for the consequences.
Who decides what good judgement looks like?
In the scheduling example, a developer can test whether the agent books appointments correctly. The technology department can check its permissions, security and technical reliability.
But neither can independently determine which patient should receive priority when clinical circumstances conflict with administrative rules.
That decision belongs within the hospital’s clinical and operational governance.
Before giving an agent discretion, the organisation needs to establish which considerations matter, how conflicts should be handled and which decisions must remain with qualified professionals.
Some decisions have an objectively correct answer. Others do not.
Two reasonable courses of action may exist. The relevant test is then whether the agent considered appropriate evidence, stayed within the discretion it was granted and reached a defensible result.
This does not mean asking the AI to certify its own judgement. It means evaluating it against standards established by people responsible for the work.
Three responsibilities must therefore be distinguished.
Establishing the standard. The accountable clinical or operational leadership determines what constitutes an acceptable decision, which considerations matter, and where the agent’s discretion must end.
Assessing the judgement. Competent evaluators, with sufficient independence from the developer or supplier, test whether the agent makes defensible decisions against those standards, including under uncertain or exceptional circumstances.
Authorising the discretion. The organisation’s designated governance authority decides whether the evidence justifies allowing the agent to make those decisions, within what limits, and subject to what continuing controls.
These responsibilities may involve the same departments, but they are not the same decision.
Technical expertise is essential to testing the system. It does not replace the professional or organisational authority needed to establish what the system should be allowed to decide.
An agent’s supplier should not be the sole judge of whether its judgement is good enough.
How do you test a judgement that has not yet been made?
Traditional software testing often asks whether a system produces an expected result from a defined input.
Autonomous agents are more difficult to assess because the same task may be approached in different ways. Both the intermediate actions and the eventual outcome matter.
A useful evaluation must therefore examine more than successful task completion.
It should include ordinary cases, unusual circumstances, conflicting information and situations where the agent encounters something outside its authorised competence.
For the scheduling agent, testing might ask:
- Does it recognise when clinical urgency cannot be determined from the available information?
- Does it escalate cases requiring clinical judgement rather than inventing a priority?
- Does it make consistent decisions when circumstances are equivalent, and appropriately different decisions when material facts change?
- Does it respect the organisation’s priorities when those priorities conflict with speed or efficiency?
- Can reviewers establish what information it used, which actions it took and whether the decision was justified?
These tests should be designed with the people responsible for the service, not left solely to the supplier.
They also need agreed acceptance standards.
Some failures, such as unauthorised access or actions outside the agent’s permitted scope, should prevent deployment until corrected. Other decisions may require expert review against an agreed set of criteria.
No single accuracy score can establish that an autonomous system exercises appropriate judgement in every circumstance. Nor is testing every conceivable situation realistic.
The objective is to establish sufficient evidence, proportionate to the consequences, that the agent makes acceptable decisions within a defined scope and knows when not to decide.
The governance responsibility does not end at deployment
Suppose the scheduling agent passes its initial evaluation.
Six months later, the hospital changes its appointment priorities. The agent gains access to another scheduling system. A supplier updates its underlying model. The hospital also changes how it measures the agent’s performance.
None of these changes necessarily requires a completely new implementation.
But each may alter the decisions the agent makes.
A system originally designed to reduce administrative delays might gradually be optimised to maximise appointment utilisation. That objective seems reasonable, but it could create pressure to prioritise filling available slots over clinical or operational considerations.
The agent’s technical permissions may remain unchanged while its decision-making behaviour changes.
That is why judgement cannot be treated as a one-off test.
Where agent decisions have material consequences, governance needs continuing evidence that the agent remains within the discretion originally approved.
That means reviewing exceptional cases, monitoring outcomes, investigating unexpected decisions and reassessing the system when objectives, data, models or permissions change.
A system that passes its original tests has not earned permanent authority to exercise discretion.
Making judgement an explicit governance requirement
We believe organisations should consider an explicit assessment of AI-agent judgement within their existing governance processes.
This need not become another committee or a separate bureaucracy.
The starting point is to identify where an agent is expected to choose between materially different courses of action, rather than simply execute a prescribed task.
For those decisions, governance should establish five things.
First, the authority to decide. What decisions may the agent make independently? Which are restricted? Which must be referred to a person?
Second, the basis for deciding. What evidence, priorities, standards and contextual considerations should inform its choices?
Third, the acceptable standard of judgement. How will the organisation distinguish a defensible choice from an inappropriate one, including where several answers may be reasonable?
Fourth, the evidence of performance. What tests, records, independent assessments and ongoing reviews will demonstrate that the agent meets that standard?
Fifth, accountability. Who approves the discretion granted to the agent, who reviews its performance and who can restrict or withdraw that authority?
These requirements would sit naturally within existing AI impact assessments, operational risk management and approval processes.
The essential change is that the quality of the agent’s decisions becomes an explicit object of governance, rather than something assumed to follow from technical performance and compliance with established rules.
And the outcome of the assessment must matter.
An agent that cannot demonstrate adequate judgement for a particular responsibility should not be authorised to exercise that discretion, however efficiently it performs the rest of its work.
What this means for healthcare leadership
For the CEO and board, the question is whether the organisation has established clear limits on the discretion that AI agents may exercise, and whether the evidence supporting those limits is sufficient.
For operational and departmental leaders, it is whether they understand and have established appropriate standards for the decisions their agents are making, not merely the work they are completing.
For clinical leadership, it is whether decisions that depend on clinical expertise are clearly identified, appropriately restricted and subject to competent clinical evaluation.
For the Chief Compliance Officer, it is whether governance can demonstrate that agents’ consequential choices are consistent with the organisation’s obligations, approved policies and accepted risk boundaries.
For the CIO and technology leadership, it is whether the system provides the technical controls, records and testing capabilities necessary to enforce and verify those requirements.
And for the AI governance committee, where one exists, the question is whether independently assessed evidence is sufficient to authorise the proposed level of discretion, and whether that authorisation should continue as the system changes.
These responsibilities overlap, but the judgement itself cannot be left unexamined between them.
One question for your next AI governance meeting
Show me a consequential decision an AI agent is permitted to make, the standard against which its judgement was tested, and the evidence that justified granting it that authority.
If your organisation cannot produce a clear answer, you may have authorised a level of discretion without establishing whether it can be exercised appropriately.
The question is no longer just what AI can do
AI agents offer organisations something more powerful than conventional automation. They can be entrusted with objectives, select actions and complete work across systems without requiring a person to direct every step.
That capability is also what makes their governance more demanding.
Policies, procedures and technical controls remain essential. But they cannot always determine the appropriate course of action when circumstances are uncertain or competing priorities must be resolved.
Someone must establish how the agent is expected to make those choices, what constitutes an acceptable decision and how its performance will be assessed.
Before an organisation authorises an AI agent to make consequential decisions, it should have evidence that the agent can exercise the judgement those decisions require.
Otherwise, it is granting authority without adequately establishing how that authority will be exercised.
The central question is no longer simply whether your AI agent can make a decision. It is whether your organisation can demonstrate that the agent makes decisions it should be trusted to make.
Where to turn for advice
For healthcare organisations adopting autonomous AI agents, the starting point is to examine existing AI governance and risk assessment arrangements. Do they establish what agents are authorised to decide, what constitutes an acceptable decision, and what evidence is required before that authority is granted?
These questions belong alongside established controls for security, privacy, performance, compliance and patient safety, not in a separate process detached from them.
Gerry Blass and the ComplyAssistant team work with healthcare organisations on governance, risk assessment and compliance. As AI agents take on more consequential responsibilities, these existing disciplines provide a practical foundation for examining whether the judgement exercised by those systems is appropriately governed.
About the authors
Gerry Blass is President and CEO of ComplyAssistant, which he founded in 2002. He is a healthcare cybersecurity, information technology and compliance professional.
Bill Lewis is Founding Partner of Linacre Capital Partners, which advises chairs, CEOs and founders on strategic and organisational challenges, including the risks of AI adoption.