Relevant powerful agents will be highly optimized

The prob­a­bil­ity that an agent that is cog­ni­tively pow­er­ful enough to be rele­vant to ex­is­ten­tial out­comes, will have been sub­ject to strong, gen­eral op­ti­miza­tion pres­sures. Two (dis­junc­tive) sup­port­ing ar­gu­ments are that, one, prag­mat­i­cally ac­cessible paths to pro­duc­ing cog­ni­tively pow­er­ful agents tend to in­voke strong and gen­eral op­ti­miza­tion pres­sures, and two, that cog­ni­tively pow­er­ful agents would be ex­pected to ap­ply strong and gen­eral op­ti­miza­tion pres­sures to them­selves.

An ex­am­ple of a sce­nario that negates Rele­van­tPow­er­fulA­gentsHigh­lyOp­ti­mized is KnownAl­gorith­mNon­re­cur­siveIn­tel­li­gence, where a cog­ni­tively pow­er­ful in­tel­li­gence is pro­duced by pour­ing lots of com­put­ing power into known al­gorithms, and this in­tel­li­gence is then some­how pro­hibited from self-mod­ifi­ca­tion and the cre­ation of en­vi­ron­men­tal sub­agents.

Whether a cog­ni­tively pow­er­ful agent will in fact have been suffi­ciently op­ti­mized de­pends on the dis­junc­tion of:

  • Pow­er­fulA­gen­tAlreadyOp­ti­mized. Fil­ter­ing for agents which are pow­er­ful enough to be rele­vant to ad­vanced agent sce­nar­ios, will in prac­tice leave mainly agents that have already been sub­jected to large amounts of cog­ni­tive op­ti­miza­tion from some in­ter­nal or ex­ter­nal source.

  • Pow­er­fulA­gen­tSelfOp­ti­mizes. Agents that reach a thresh­old of power suffi­cient to be rele­vant to ad­vanced agent sce­nar­ios will af­ter­wards op­ti­mize them­selves both heav­ily and gen­er­ally.

End­ing up with a sce­nario along the lines of KnownAl­gorith­mNon­re­cur­siveIn­tel­li­gence re­quires defeat­ing both of the above con­di­tions si­mul­ta­neously. The sec­ond con­di­tion seems more difficult and to re­quire more Cor­rigi­bil­ity or Ca­pa­bil­i­tyCon­trol fea­tures than the first.


  • AI alignment

    The great civ­i­liza­tional prob­lem of cre­at­ing ar­tifi­cially in­tel­li­gent com­puter sys­tems such that run­ning them is a good idea.