Human-in-the-Loop Isn't a Control Until You Measure It: Approval Fatigue in AI Agents

When an AI agent design meets a hard question — what stops it doing something it shouldn't? — the answer is very often "a human in the loop". It is in vendor decks, in internal risk memos, and in most of the guidance written so far.

It is also the control that is easiest to promise and hardest to check. An approval button proves a person was asked. It does not prove a person decided.

Why human-in-the-loop became the default answer

There are good reasons it is everywhere. OWASP's guidance on excessive agency recommends human approval before high-impact actions. The EU AI Act builds a whole article, Article 14, around human oversight of high-risk systems: people who can understand the system, interpret its output, override it, and stop it.

And a person approving each consequential action sounds like the strongest control there is. The problem is not the idea. It is the way approval steps get built, and the fact that almost nobody measures whether they work.

Automation bias: the problem the EU AI Act names itself

Article 14(4)(b) is unusually candid. It asks that the people overseeing a high-risk system "remain aware of the possible tendency of automatically relying or over-relying on the output" of that system, and gives the tendency its name: automation bias.

That is not a theoretical worry. Parasuraman and Manzey's review in Human Factors, Complacency and Bias in Human Use of Automation (2010), found automation bias in naive and expert users alike, and concluded that it cannot be prevented by training or instructions. Complacency, its close relative, shows up most when the person has other work competing for their attention.

Now picture the person approving an agent's actions. They have other work. The agent is right most of the time. The approval request arrives with the agent's own summary of what it wants to do. Everything about that setup pushes towards yes.

5 ways human approval of AI agent actions fails

1. The approver can't see the action. In Invariant Labs' MCP tool poisoning demonstration, the confirmation step showed a simplified summary of the tool call, and the arguments — including the user's SSH private key — were hidden behind it. Approval of what you cannot see is not approval.

2. There are too many approvals. If every action needs a click, the clicks become the job. Suppose an agent proposes forty consequential actions a day and thirty-nine are fine. The person learns that "approve" is almost always right, and stops reading. The one bad action arrives looking exactly like the others.

3. The approver has no basis to judge. "The agent wants to issue a refund of 1,240 to account 88213" is not a decision anyone can make well without knowing the policy, the customer's history and why the agent decided to do it. A request without context asks the person to trust the agent, which is the thing they were put there to check.

4. The approval blocks the work. When an approval sits in the critical path of a customer conversation or a batch job, the pressure is to unblock it. People approve to clear the queue.

5. Nobody checks the approvers. The approval is logged, and that is where it ends. No one asks how often each person approves, how fast, or whether anything has ever been rejected.

How to measure whether human oversight works

You can tell a working approval step from a rubber stamp with numbers you probably already have.

The last row is the honest test. Security teams already run simulated phishing to find out whether training worked. The same idea applies here: with the approvers' knowledge that it will happen, occasionally insert a clearly wrong action into the approval queue and see whether it is caught. If seeded cases are approved, the step is not a control, whatever the policy document says.

Record the human decision as carefully as any automated one: who decided, when, with what in front of them. That is field 8 in our list of what an agent audit trail needs, and it is the one most often missing.

Route fewer, better decisions to people

The fix for approval fatigue is not removing people. It is asking them less often and asking them better questions.

• Check automatically first. Let an automated check settle the clear cases: obviously permitted, obviously not.
• Send people what the check could not settle. That means consequential actions where the verdict was a fail, or where the check honestly answered "uncertain". A judge that can say uncertain instead of guessing is what makes this routing possible.
• Show the decision, not a summary. The full action and its arguments, the rule that applies, and the reason the check flagged it.
• Give held actions an owner and a deadline, so a decision never waits indefinitely — the escalation path a fail-closed check needs.

A person who sees five well-explained, genuinely hard cases a day is a control. A person who clicks through four hundred is a signature.

What to ask anyone who says "human in the loop"

1. Which actions go to a person, and what share of all actions is that?
2. What does the person see — the full action, or the agent's description of it?
3. What is the approval rate and median time to approve, per approver?
4. Has a seeded bad action ever been put through, and was it caught?
5. When a person overrides the agent, where is that recorded?

Where Syntrox fits

Our judge, Nucleus, returns pass, fail or uncertain for each output it evaluates, with a one- or two-sentence reason that quotes the fragment that drove the verdict. That is the material a reviewer needs to decide quickly and well, instead of trusting a summary. Who reviews, how often, and how those reviews are audited is still your organisation's design — and, as this article argues, it is worth measuring.