You can't get the coffee if you're dead

“You can’t get the coffee if you’re dead” is Stu­art Rus­sell’s cap­sule de­scrip­tion of why al­most any agent goal im­plies an in­stru­men­tal strat­egy of sur­viv­ing /​ not be­ing shut down—by de­fault and bar­ring other mea­sures to pre­vent this from hap­pen­ing; and as­sum­ing the agent has suffi­cient big-pic­ture aware­ness to un­der­stand the re­la­tion be­tween the agent’s non-op­er­a­tion and the coffee not be­ing brought. See Shut­down prob­lem and In­stru­men­tal con­ver­gence.


  • Shutdown problem

    How to build an AGI that lets you shut it down, de­spite the ob­vi­ous fact that this will in­terfere with what­ever the AGI’s goals are.