Sufficiently advanced Artificial Intelligence

A ‘suffi­ciently ad­vanced Ar­tifi­cial In­tel­li­gence’ is a cog­ni­tive agent or cog­ni­tive al­gorithm with ca­pa­bil­ities great enough that we need to think about it in a qual­i­ta­tively differ­ent way from robotic cars; a ma­chine in­tel­li­gence smart enough that we need to start do­ing AI al­ign­ment the­ory to it.

For ex­am­ple, we prob­a­bly don’t need to worry about an AI that tries to pre­vent us from press­ing its off-switch, un­til the AI knows that (a) it has an off-switch and (b) press­ing the off-switch will pre­vent the AI from achiev­ing its other goals.

In turn, this knowl­edge is a spe­cial case of Big-pic­ture strate­gic aware­ness, which might fol­low from learn­ing many facts about many do­mains via Ar­tifi­cial Gen­eral In­tel­li­gence.

The page on ad­vanced agent prop­er­ties starts to list out some of the differ­ent ways that an AI could be ‘smart enough’ in this sense, along with the par­tic­u­lar prob­lems that might be en­coun­tered with an AI that smart.


  • Advanced agent properties

    How smart does a ma­chine in­tel­li­gence need to be, for its nice­ness to be­come an is­sue? “Ad­vanced” is a broad term to cover cog­ni­tive abil­ities such that we’d need to start con­sid­er­ing AI al­ign­ment.