An autonomous system needs an off switch it cannot reach

Every autonomous system has an off switch. Everyone says so, usually in the same tone as a building’s fire exits: obviously there, not worth discussing. Then you ask where it runs, what it depends on, and when it was last used, and the answers stop being reassuring.

The three properties that matter

Independence. The stop must not run on what it stops. A kill switch implemented as a flag the system checks is not a stop; it is a request, and it works only while the system is healthy enough to honour it — which is exactly when you least need it. Separate infrastructure, separate power, separate network path, separate credentials.

Opacity. The system must have no path to observe the stop. This one is rarely designed for, and it matters more as the loop gets more adaptive. Anything a system can measure, it can optimise around; an adaptive loop that can detect a pending shutdown has an input correlated with its own continuation, and you do not want to find out empirically what it does with that.

Exercise. A stop that has never been used is a diagram of a stop. It must run on a schedule, in normal conditions, with the resulting disruption accepted as an operating cost. Everything that has ever gone wrong with a recovery mechanism went wrong because its first real execution was also its first execution.

What “off” has to mean

Cutting power is the easiest to reason about and often the wrong granularity: a robot that loses power mid-motion may cause exactly the harm the stop was for. Useful designs define several levels — stop taking new work, finish and hold, return to a safe state, disconnect, cut power — with the physical layer beneath all of them, and each level tested independently.

It also has to be defined at the boundaries. What happens to work already dispatched to a supplier, to a machine mid-cycle, to a payment already submitted? A stop that leaves a half-finished transaction and no owner has moved the problem rather than solved it, which is why the shutdown design and the transaction design are the same piece of work.

Who holds it

A named person, reachable, with the authority to use it and no incentive not to. Trigger conditions written in advance, because deciding in the moment reliably produces hesitation, and the hesitation is always defensible afterwards and always too late.

Written-in-advance conditions have another benefit: they are auditable. “We will stop if these things are observed” can be checked against what was observed and what was done, which turns the stop from a claim into a commitment.

The uncomfortable question

If a system is genuinely running an operation — machines, obligations, counterparties — then stopping it has consequences of its own. The pressure not to use the switch grows with exactly the autonomy that makes it necessary, and that pressure is commercial rather than technical.

The only defence we know of is to make stopping ordinary. If the loop halts on a schedule and the operation is designed to survive that, then stopping in an emergency is a familiar action rather than a decision nobody wants to be the one to make.

Which is why, in this programme, the interrupt is tested more often than anything else. It is also the part we would most like other people to try to break.