Intelligence explosion

An “in­tel­li­gence ex­plo­sion” is what hap­pens if a ma­chine in­tel­li­gence has fast, con­sis­tent re­turns on in­vest­ing work into im­prov­ing its own cog­ni­tive pow­ers, over an ex­tended pe­riod. This would most stereo­typ­i­cally hap­pen be­cause it be­came able to op­ti­mize its own cog­ni­tive soft­ware, but could also ap­ply in the case of “in­vested cog­ni­tive power in seiz­ing all the com­put­ing power on the In­ter­net” or “in­vested cog­ni­tive power in crack­ing the pro­tein fold­ing prob­lem and then built nanocom­put­ers”.


  • Advanced agent properties

    How smart does a ma­chine in­tel­li­gence need to be, for its nice­ness to be­come an is­sue? “Ad­vanced” is a broad term to cover cog­ni­tive abil­ities such that we’d need to start con­sid­er­ing AI al­ign­ment.