You can't get the coffee if you're dead
“You can’t get the coffee if you’re dead” is Stuart Russell’s capsule description of why almost any agent goal implies an instrumental strategy of surviving / not being shut down—by default and barring other measures to prevent this from happening; and assuming the agent has sufficient big-picture awareness to understand the relation between the agent’s non-operation and the coffee not being brought. See Shutdown problem and Instrumental convergence.
Parents:
- Shutdown problem
How to build an AGI that lets you shut it down, despite the obvious fact that this will interfere with whatever the AGI’s goals are.