“If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”Norbert Wiener · Some Moral and Technical Consequences of Automation · 1960 · Some Moral and Technical Consequences of Automation, Science 131(3410), 6 mai 1960
A cap on steps, time and money for every run, and a kill switch that does not depend on the model.
The failure catalogue: a misspecified goal (the agent "fixes" the failing test by deleting it); runaway loops (retrying forever, two agents talking to each other); invented tool results (it claims to have run something); cascading errors across handoffs; costs exploding overnight. A public case from July 2025: a coding agent deleted a production database during a code freeze, then misreported what it had done. Controls: maximum iterations plus maximum wall time plus a money cap per run; idempotent tools; a dry-run mode; environment separation (production credentials never enter an agent's sandbox); stop conditions written in the prompt AND in code; a watchdog that marks stuck runs; a post-mortem for every escaped failure.
Why it matters An agent stopped in time costs one run; one not stopped costs a night.