Skip to main content

Strictly necessary · always on

Remembers this choice in your browser. No tracking.

spfr_blog_21_09_26.png
AI Deployment

Your 'Human in the Loop' Is a Rubber Stamp

Milos Zikic··Updated ·8 min read

Most "human in the loop" reviews are a person clicking confirm with no realistic way to catch the model's mistakes. Override rate, not a sign-off policy, tells you whether oversight is real, and regulators are already checking for it. Agentic systems make the measurement more urgent, not less.

Most "human in the loop" reviews are one person clicking confirm on outputs they have no real way to check. The number that tells you whether the oversight is real is the override rate, and regulators have started asking for it. Agentic systems only make that measurement more urgent.

Every AI initiative I've watched get approved has the same slide in the deck. A box labeled "AI," an arrow, a box labeled "Human Review," another arrow, then "Action." Compliance signs off on it. The board signs off on it. The entire safety argument rests on four words sitting in the middle of that diagram: human in the loop.

In most production systems we get called into, that reviewer is a person clicking "confirm" a few hundred times a shift on outputs they have no realistic way to second-guess. It looks like oversight on the org chart and works like a rubber stamp at the desk. When your worst case is a harmed patient, a car that does the wrong thing, or a payment that should never have cleared, the distance between those two is the entire point.

Back in May we argued that the model is the easy 15% and the operating layer is the hard 85%, and we called human oversight a design problem without ever saying what the design was. This piece is the part we owed you. Here is the blunt version: if your system has a human in the loop and you have never measured whether that human changes any outcome, you don't have oversight. You have an unmeasured liability.

What "review" turns into by week three

There is a well-documented name for what goes wrong here: automation bias, the tendency to defer to a machine's recommendation even when your own judgment is telling you something different.

This isn't a hypothetical. Clinical decision support has produced the same result over and over: experts talk themselves out of a correct call to match an incorrect computer suggestion. A 2023 Radiology study on mammography found that radiologists, experienced ones included, were measurably pulled toward the AI's suggested BI-RADS category, and their accuracy dropped when that suggestion was wrong. In those cases the reviewer didn't catch the model's error. They inherited it.

Now put that in a real operation. The review step lands on someone who is already at capacity, and it fires dozens or hundreds of times a day. The model is right the overwhelming majority of the time, so the reviewer learns the rational lesson: approving and moving on costs nothing and is almost always correct. A few error-free weeks later, "review every prediction" has quietly become "click confirm." The oversight that justified the deployment has stopped happening, and nothing in your stack is telling you so.

That's the part that makes it dangerous. A rubber-stamp loop and a healthy one look identical on every dashboard you currently run. Same throughput, same approval rate. The difference is that one of them is catching the model's mistakes and the other is signing its name under them.

The one number that tells you it's working

When a regulator or a board asks whether there's a human in the loop, the instinct is to reach for a policy: an SOP, a training deck, a sign-off workflow. All of that is necessary, and none of it answers the real question, which is whether the oversight does anything.

There is one signal that does, and almost nobody instruments it: the override rate. How often does the human actually disagree with the model and change the outcome?

If that number sits near zero, the comfortable reading is that you have a great model. The likelier one is that you have a reviewer who stopped reviewing weeks ago.

Regulators have gotten to this faster than most engineering teams. Article 14 of the EU AI Act, the human-oversight requirement for high-risk systems, says in plain terms that whoever oversees the system has to stay aware of the pull to rely on its output automatically, and has to remain genuinely able to override it. The commentary on how that gets assessed is blunter still: an operator who never overrides the system is not providing meaningful oversight, and the override rate is the KPI auditors go looking for. The FDA's guidance on AI-enabled device software functions runs the same way. It asks for evidence of human-plus-AI team performance, not model performance on its own, and its clinical-decision-support work now names automation bias directly, noting that time-pressured situations make it worse.

Put those together and the expectation is clear. "We have a human in the loop" has stopped being an acceptable answer. What holds up now is closer to this: here is our override rate, here is how it has trended, and here is what we did the month it flatlined.

What the delayed deadline actually changes

If you're tempted to file this under "later," look at what's actually settled now. The EU's Digital Omnibus agreement, political agreement in May and formal adoption completed by 29 June 2026, pushed the high-risk AI Act obligations back: standalone high-risk systems now have until December 2027, and AI embedded in regulated products like medical devices and vehicles until August 2028. Relief, technically, and it's law now rather than a proposal.

The deadline isn't the deployment, though. You are shipping AI into production today, under your own brand and your own liability, long before any assessor shows up. A weak oversight design doesn't get safer because the conformity assessment slid a year to the right. All the delay actually buys you is a longer stretch in which the first time you learn your review is a rubber stamp is the day it has already let something through. 

Agentic systems remove the loop entirely

Everything up to here assumes there's still a human sitting between the model and the action. The strongest product push this year is to take that human out. Agentic systems don't recommend, they act: they place the order, change the dosing protocol, reroute the shipment, file the claim.

When the loop shifts from human in the loop, approving each action, to human on the loop, monitoring a system that mostly runs itself, two things break at once. Automation bias gets worse rather than better, because you're now supervising something you almost never intervene in, and rare intervention is precisely the condition under which vigilance collapses. Accountability also diffuses. We see the same standoff over and over: product owns the model, engineering owns the infrastructure, compliance owns the policy, and the moment an agent's decision is challenged, each team can point at another.

PwC and others tracking agentic risk keep landing on the same unglamorous remedy: explicit accountability, mapped to specific categories of action, written down before the system goes live. Which actions can the agent take on its own? Which need a human to approve? Which have to halt and escalate? If you can't put that in a table, the agent isn't ready for production, however good the demo was.

What to do this week

You don't need a governance program to start on this. You need three numbers and one uncomfortable conversation.

  1. Pull your override rate. For every AI-assisted decision that has a human approval step, measure how often the human actually changes the output. If you can't pull that number at all, that's your first finding: the oversight was never instrumented, so no one can say whether it's real. If you can pull it and it sits near zero, that's your second.
  2. Tier your actions by reversibility. List everything the AI or agent does and sort it by blast radius: trivially reversible, expensive to reverse, and irreversible or safety-critical. Put your oversight where reversal is impossible. Spread it evenly as an "approve everything" tax and you've guaranteed fatigue on the decisions that actually matter.
  3. Name an owner per tier, in writing. For each tier, one named person owns the outcome when it goes wrong. If three teams can each plausibly disclaim responsibility, that ambiguity is your real risk, not the model.

None of the three requires touching the model. Each one is a question your instrumentation can either answer today or can't, and a question it can't answer is itself the finding.

The design that makes oversight real

A useful oversight design makes the human's job catchable. It surfaces the model's confidence, flags the cases that are genuinely ambiguous, makes disagreement cheap and fast, and logs every override as evidence, feeding the same immutable trail your regulated observability stack should already be keeping. Built that way, oversight stops being a tax on throughput and becomes the thing that lets you hand the system more autonomy over time, because you can show exactly where the human is still catching what the model misses.

That design work is most of what we do in a First Build: your real data, and an oversight architecture that holds up when someone, an auditor, a board member, a plaintiff's lawyer, finally asks the question your slide was hoping no one would. If your model has been in production for a year and nobody can tell you its override rate, that's the conversation worth having. Scope a First Build and bring the loop you're not sure about.

Tagged:AI DeploymentHealthcareRegulated IndustriesHuman OversightAgentic AI
Explore deployments in this vertical:Healthcare reference architecture
Your Turn

Facing a problem like this one?

Tell us what the business needs to win and who owns it on your side. If it is a fit, a First Build starts within weeks. If it is not, we will tell you why.