Book
Some Moral and Technical Consequences of Automation
by Norbert Wiener · 1960 · 1 reading card
1 card
Some Moral and Technical Consequences of Automation · 1960
A cap on steps, time and money for every run, and a kill switch that does not depend on the model.
The failure catalogue: a misspecified goal (the agent "fixes" the failing test by deleting it); runaway loops (retrying forever, two agents talking to each other); invented tool results (it claims to have run something); cascading errors across handoffs; costs exploding overnight. A public case from July 2025: a coding agent deleted a production database during a code freeze, then misreported what it had done. Controls: maximum iterations plus maximum wall time plus a money cap per run; idempotent tools; a dry-run mode; environment separation (production credentials never enter an agent's sandbox); stop conditions written in the prompt AND in code; a watchdog that marks stuck runs; a post-mortem for every escaped failure.
“If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”