Relevant powerful agents will be highly optimized
The probability that an agent that is cognitively powerful enough to be relevant to existential outcomes, will have been subject to strong, general optimization pressures. Two (disjunctive) supporting arguments are that, one, pragmatically accessible paths to producing cognitively powerful agents tend to invoke strong and general optimization pressures, and two, that cognitively powerful agents would be expected to apply strong and general optimization pressures to themselves.
An example of a scenario that negatesis , where a cognitively powerful intelligence is produced by pouring lots of computing power into known algorithms, and this intelligence is then somehow prohibited from self-modification and the creation of environmental subagents.
Whether a cognitively powerful agent will in fact have been sufficiently optimized depends on the disjunction of:
. Filtering for agents which are powerful enough to be relevant to advanced agent scenarios, will in practice leave mainly agents that have already been subjected to large amounts of cognitive optimization from some internal or external source.
. Agents that reach a threshold of power sufficient to be relevant to advanced agent scenarios will afterwards optimize themselves both heavily and generally.
Ending up with a scenario along the lines of Corrigibility or features than the first.requires defeating both of the above conditions simultaneously. The second condition seems more difficult and to require more
- AI alignment
The great civilizational problem of creating artificially intelligent computer systems such that running them is a good idea.