When an AI agent should stop and fetch a human

Escalation is the design, not the fallback

Most agent projects design the happy path in detail and treat escalation as an error state to be added later. That ordering is backwards, and you can usually tell within ten minutes of reading a spec which way round a team has done it.

The reason is that the happy path is the part you can improve incrementally forever, whereas the failure path determines whether anyone will leave the system running unattended. Staff do not abandon an automation because it is imperfect. They abandon it because it did something they could not see, could not explain to their manager, and could not undo. Design the moment it stops before you design the moment it works.

Trigger one: low confidence, honestly measured

The obvious trigger is the agent not being sure. The difficulty is that language models are poorly calibrated by default: they produce fluent, confident text for things they have no basis for, and the confidence expressed in the prose has almost no relationship to whether the content is correct.

So confidence has to be something you construct, never something you read off. In practice that means grounding the answer in a retrievable source and escalating when nothing relevant comes back, checking whether the same question answered twice produces the same result, and treating any required field the agent could not fill from a real system as a stop condition instead of something to infer.

The cheapest version of this, and the one I reach for first, is a required-evidence rule: the agent may only act on facts it can point at. If it cannot cite the record it took a value from, it does not get to use that value. This eliminates a large class of confident invention without any calibration machinery at all.

Trigger two: asymmetric cost

Some mistakes are cheap to undo and some are not, and the agent should be told which is which explicitly, because it has no way of working it out. Sending a slightly awkward acknowledgement email is recoverable. Issuing a refund, cancelling a booking somebody travelled for, deleting a record or promising a delivery date are not, or not without a person spending an afternoon on it.

The rule I use is blunt: if undoing an action costs more than doing it, a person authorises it. That single line removes most of the decisions people are actually frightened of, and it does so without needing anyone to agree on how good the model is.

It also handles a case teams forget. Asymmetry is not always financial. A tone-deaf reply to a bereaved customer costs nothing to send and a great deal to have sent, and any agent touching sensitive categories should route those to a person regardless of how confident it is.

Trigger three: nobody asked it to do this

The third trigger is the one that gets skipped, and it is the one that produces the incidents. An agent given tools will try to use them on whatever arrives, including requests nobody considered when it was built. A booking agent asked a billing question will attempt the billing question, because nothing told it not to.

The fix is boring and effective: the list of what it handles is written down, the agent is told what falls outside it, and anything it does not recognise goes to a person by default. Deny by default is old security practice and it applies here unchanged. The cost is some unnecessary hand-offs in the first fortnight, which tells you where the list needs sharpening.

What a good handover actually contains

An escalation that arrives as a notification saying the agent needs help is not an escalation, it is an interruption. The person picking it up has to reconstruct everything the agent already knew, which is slower than if the agent had never been involved, and after a fortnight of that the team will route around the system.

A handover worth building carries five things, and it is not a long list:

• What was being attempted, in one line a colleague can read.
• What the agent already established, with links to the records it read.
• Why it stopped, naming which of the three triggers fired.
• What it has already done, so nobody repeats a step or sends a second email.
• Who owns it now, by name, with the clock started.

What escalation is not

It is not a queue with no owner. Routing to a shared inbox that four people can see and nobody is responsible for is how an escalation path becomes a place work goes to die, and it will still look green on every dashboard you have.

It is not an apology and a phone number, either. Telling a caller that somebody will be in touch, with no slot booked and no context saved, moves the problem to them. If the agent can offer a real time from a real calendar, it should, and the person who takes that call should arrive knowing what has already happened.

The escalation rate is a dial, not a score

Teams often treat the proportion of escalated cases as a grade, and try to drive it toward zero. It is better understood as a dial you set deliberately, high at launch and lowered as evidence accumulates about which cases the agent handles well.

Two numbers are worth watching more than the rate itself. The first is how many escalations turn out to have been unnecessary, which tells you the threshold is too tight. The second, and the one that matters, is how many incidents were found by somebody other than the system: a customer complaining, a colleague noticing a wrong figure. Any of those means the agent failed and did not know it, which is the only failure mode on this page worth losing sleep over.