Human in the loop: how to deploy AI agents without losing control

Every week I talk to someone who either won’t let AI near their business or has already let it loose on their inbox. Both are mistakes. The first leaves real time on the table. The second is how you end up apologising to customers.

The middle path is simple to describe: give agents the autonomy their track record has earned, and no more.

Start with the cost of a mistake

Before deciding how much freedom an agent gets, ask two questions about the action it takes:

1. How bad is it if this goes wrong? A mislabelled ticket is an annoyance. A wrong refund is money. A rude email to your biggest client is a relationship.
2. Can it be undone? A draft can be deleted. A sent email, a payment or a deleted record often can’t.

Cheap and reversible actions can be automated early. Expensive or irreversible ones stay with people for a long time, possibly forever.

The autonomy ladder

I use four levels. Every task an agent performs sits on one of them, and tasks move up one level at a time.

Different tasks inside the same agent can sit at different levels. An intake agent might sort enquiries at level 3, reply to standard questions at level 2 and send quotes only at level 1.

Actions that should always involve a person

Some decisions should keep a human in the loop no matter how good the agent gets:

• Moving money: refunds, payments, discounts and credit.
• Deleting or overwriting data that can’t be restored.
• Pricing and contractual commitments outside published rules.
• Regulated advice: legal, medical, financial or tax guidance.
• Anything reputational: replies to complaints, public posts, press.

This isn’t about distrust. It’s about accountability: when something goes wrong, a person needs to own the decision.

Guardrails that belong in every build

Approval steps are one safety layer. These are the others I put into every agent:

• Least privilege. The agent can only reach the tools and records its job needs. Read-only unless writing is essential.
• Hard limits. Caps on how many messages it can send per hour, how much it can spend and which recipients it can contact.
• Confidence thresholds. When the agent is unsure, or data doesn’t reconcile, it stops and asks instead of guessing.
• Complete logs. Every input, decision and action is recorded, so any outcome can be traced and explained.
• A kill switch. One obvious way to pause everything, known to the whole team.
• Honest disclosure. Customers are told when an automated assistant is involved, with an easy route to a person.

Roll out in shadow mode

Launch every new agent at level 0 or 1. For the first week or two, it drafts and a person reviews. Measure three things:

• Approval rate without edits: how often the draft was good enough to send as-is.
• Correct escalations: how often it flagged the cases that genuinely needed a person.
• Escaped errors: mistakes that reached a customer despite review.

When the numbers are consistently strong, promote the routine part of the task to level 2. Keep measuring.

The bottom line

Control isn’t the enemy of automation. It’s what makes automation safe to scale. Decide autonomy by the cost and reversibility of mistakes, climb the ladder one step at a time, and keep people firmly in charge of money, deletions, commitments and anything that could hurt a relationship. If you’re choosing your first agent task, the AI time audit is the place to start.