General intelligence


Although hu­mans share 95% of their DNA with chim­panzees, and have brains only three times as large as chim­panzee brains, hu­mans ap­pear to be far bet­ter than chim­panzees at learn­ing an enor­mous va­ri­ety of cog­ni­tive do­mains. A bee is born with the abil­ity to con­struct hives; a beaver is born with an in­stinct for build­ing dams; a hu­man looks at both and imag­ines a gi­gan­tic dam with a hon­ey­comb struc­ture of in­ter­nal re­in­force­ment. Ar­guendo, some set of fac­tors, pre­sent in hu­man brains but not in chim­panzee brains, seem to sum to a cen­tral cog­ni­tive ca­pa­bil­ity that lets hu­mans learn a huge va­ri­ety of differ­ent do­mains with­out those do­mains be­ing speci­fi­cally pre­pro­grammed as in­stincts.

This very-widely-ap­pli­ca­ble cog­ni­tive ca­pac­ity is termed gen­eral in­tel­li­gence (by most AI re­searchers ex­plic­itly talk­ing about it; the term isn’t uni­ver­sally ac­cepted as yet).

We are not perfectly gen­eral—we have an eas­ier time learn­ing to walk than learn­ing to do ab­stract calcu­lus, even though the lat­ter is much eas­ier in an ob­jec­tive sense. But we’re suffi­ciently gen­eral that we can figure out Spe­cial Rel­a­tivity and en­g­ineer skyscrap­ers de­spite our not hav­ing those abil­ities built-in at com­pile time (i.e., at birth). An Ar­tifi­cial Gen­eral In­tel­li­gence would have the same prop­erty; it could learn a tremen­dous va­ri­ety of do­mains, in­clud­ing do­mains it had no inkling of when it was switched on.

More spe­cific hy­pothe­ses about how gen­eral in­tel­li­gence op­er­ates have been ad­vanced at var­i­ous points, but any cor­re­spond­ing at­tempts to define gen­eral in­tel­li­gence that way, would be the­ory-laden. The prethe­o­ret­i­cal phe­nomenon to be ex­plained is the ex­traor­di­nary va­ri­ety of hu­man achieve­ments across many non-in­stinc­tual do­mains, com­pared to other an­i­mals.

Ar­tifi­cial Gen­eral In­tel­li­gence is not par-hu­man AI

Since we only know about one or­ganism with this ‘gen­eral’ or ‘sig­nifi­cantly more gen­er­ally ap­pli­ca­ble than chim­panzee cog­ni­tion’ in­tel­li­gence, this ca­pa­bil­ity is some­times iden­ti­fied with hu­man­ity, and con­se­quently with our over­all level of cog­ni­tive abil­ity.

We do not, how­ever, know that “cog­ni­tive abil­ity that works on a very wide va­ri­ety of prob­lems” and “over­all hu­man­ish lev­els of perfor­mance” need to go to­gether across much wider differ­ences of mind de­sign.

Hu­mans evolved in­cre­men­tally out of ear­lier ho­minids by blind pro­cesses of nat­u­ral se­lec­tion; evolu­tion wasn’t try­ing to de­sign a hu­man on pur­pose. Be­cause of the way we evolved in­cre­men­tally, all neu­rotyp­i­cal hu­mans have spe­cial­ized evolved ca­pa­bil­ities like ‘walk­ing’ and ‘run­ning’ and ‘throw­ing stones’ and ‘out­wit­ting other hu­mans’. We have all the pri­mate ca­pa­bil­ities and all the ho­minid ca­pa­bil­ities as well as what­ever is strictly nec­es­sary for gen­eral in­tel­li­gence.

So, for all we know at this point, there could be some way to get a ‘sig­nifi­cantly more gen­eral than chim­panzee cog­ni­tion’ in­tel­li­gence, in the equiv­a­lent of a weaker mind than a hu­man brain. E.g., due to leav­ing out some of the spe­cial sup­port we evolved to run, throw stones, and out­wit other minds. We might at some point con­sis­tently see an in­frahu­man gen­eral in­tel­li­gence that is not like a dis­abled hu­man, but rather like some pre­vi­ously un­ob­served and uni­mag­ined form of weaker but still highly gen­eral in­tel­li­gence.

Since the con­cepts of ‘gen­eral in­tel­li­gence’ and ‘roughly par-hu­man in­tel­li­gence’ come apart in the­ory and pos­si­bly also in prac­tice, we should avoid speak­ing of Ar­tifi­cial Gen­eral In­tel­li­gence as if were iden­ti­cal with a con­cept like “hu­man-level AI”.

Gen­eral in­tel­li­gence is not perfect intelligence

Gen­eral in­tel­li­gence doesn’t im­ply the abil­ity to solve ev­ery kind of cog­ni­tive prob­lem; if we wanted to use a longer phrase we could say that hu­mans have ‘sig­nifi­cantly more gen­er­ally ap­pli­ca­ble in­tel­li­gence than chim­panzees’. A suffi­ciently ad­vanced Ar­tifi­cial In­tel­li­gence that could self-mod­ify (rewrite its own code) might have ‘sig­nifi­cantly more gen­er­ally ap­pli­ca­ble in­tel­li­gence than hu­mans’; e.g. such an AI might be able to eas­ily write bug-free code in virtue of giv­ing it­self spe­cial­ized cog­ni­tive al­gorithms for pro­gram­ming. Hu­mans, to write com­puter pro­grams, need to adapt sa­vanna-spe­cial­ized tiger-eva­sion mod­ules like our vi­sual cor­tex and au­di­tory cor­tex to rep­re­sent­ing com­puter pro­grams in­stead, which is one rea­son we’re such ter­rible pro­gram­mers.

Similarly, it’s not hard to con­struct math prob­lems to which we know the solu­tion, but are un­solv­able by any gen­eral cog­ni­tive agent that fits in­side the phys­i­cal uni­verse. For ex­am­ple, you could pick a long ran­dom string and gen­er­ate its SHA-4096 hash, and if the SHA al­gorithm turns out to be se­cure against quan­tum com­put­ing, you would be able to con­struct a highly spe­cial­ized ‘agent’ that could solve the prob­lem of ‘tell me which string has this SHA-4096 hash’ which no other agent would be able to solve with­out di­rectly in­spect­ing your agent’s cog­ni­tive state, or trick­ing your agent into re­veal­ing the se­cret, etcetera. The ‘sig­nifi­cantly more gen­er­ally ap­pli­ca­ble than chim­panzee in­tel­li­gence’ of hu­mans is able to figure out how to launch in­ter­plane­tary space probes just by star­ing at the en­vi­ron­ment for a while, but it still can’t re­verse SHA-4096 hashes.

It would how­ever be an in­stance of the con­tinuum fal­lacy, nir­vana fal­lacy, false di­chotomy, or straw su­per­power fal­lacy, to ar­gue:

  • Some small agents can solve cer­tain spe­cific math prob­lems un­solv­able by much larger su­per­in­tel­li­gences.

  • There­fore there is no perfectly gen­eral in­tel­li­gence, just a con­tinuum of be­ing able to solve more and more prob­lems.

  • There­fore there is noth­ing wor­thy of re­mark in how hu­mans are able to learn a far wider va­ri­ety of do­mains than chim­panzees, nor any sharp jump in gen­er­al­ity that an AI might ex­hibit in virtue of ob­tain­ing some cen­tral set of cog­ni­tive abil­ities.

For at­tempts to talk about perfor­mance rel­a­tive to a truly gen­eral mea­sure of in­tel­li­gence (as op­posed to just say­ing that hu­mans seem to have some cen­tral ca­pa­bil­ity which sure lets them learn a whole lot of stuff) see Shane Legg and Mar­cus Hut­ter’s work on pro­posed met­rics of ‘uni­ver­sal in­tel­li­gence’.

Gen­eral in­tel­li­gence is a sep­a­rate con­cept from IQ /​ g-factor

Charles Spear­man found that by look­ing on perfor­mances across many cog­ni­tive tests, he was able to in­fer a cen­tral fac­tor, now called Spear­man’s g, which ap­peared to be more cor­re­lated with perfor­mance on each task than any of the tasks were cor­re­lated with each other.

[For ex­am­ple](https://​​en.wikipe­​​wiki/​​G_fac­tor_(psy­cho­met­rics)), the cor­re­la­tion be­tween stu­dents’ French and English scores was 0.67: that is, 67% of the vari­a­tion in perfor­mance in French could be pre­dicted by look­ing at the stu­dent’s score in English.

How­ever, by look­ing at all the test re­sults to­gether, it was pos­si­ble to con­struct a cen­tral score whose cor­re­la­tion with the stu­dent’s French score was 88%.

This would make sense if, for ex­am­ple, the score in French was “g-fac­tor plus un­cor­re­lated vari­ables” and the score in English was “g-fac­tor plus other un­cor­re­lated vari­ables”. In this case, the set­ting of the g-fac­tor la­tent vari­able, which you could in­fer bet­ter by look­ing at all the stu­dent’s scores to­gether, would be more highly cor­re­lated with both French and English ob­ser­va­tions, than those tests would be cor­re­lated with each other.

In the con­text of Ar­tifi­cial In­tel­li­gence, g-fac­tor is not what we want to talk about. We are try­ing to point to a fac­tor sep­a­rat­ing hu­mans from chim­panzees, not to in­ter­nal vari­a­tions within the hu­man species.

That is: If you’re try­ing to build the first me­chan­i­cal heav­ier-than-air fly­ing ma­chine, you ought to be think­ing “How do birds fly? How do they stay up in the air, at all?” Rather than, “Is there a cen­tral Fly-Q fac­tor that can be in­ferred from the vari­a­tion in many differ­ent mea­sures of how well in­di­vi­d­ual pi­geons fly, which lets us pre­dict the in­di­vi­d­ual vari­a­tion in a pi­geon’s speed or turn­ing ra­dius bet­ter than any sin­gle ob­ser­va­tion about one fac­tor of that pi­geon’s fly­ing abil­ity?”

In some sense the ex­is­tence of g-fac­tor could be called Bayesian ev­i­dence for the no­tion of gen­eral in­tel­li­gence: if gen­eral in­tel­li­gence didn’t ex­ist, prob­a­bly nei­ther would IQ. Like­wise the ob­ser­va­tion that, e.g., John von Neu­mann ex­isted and was more pro­duc­tive across mul­ti­ple dis­ci­plines com­pared to his aca­demic con­tem­po­raries. But this is not the main ar­gu­ment or the most im­por­tant ev­i­dence. Look­ing at hu­mans ver­sus chim­panzees gives us a much, much stronger hint that a species’ abil­ity to land space probes on Mars cor­re­lates with that species’ abil­ity to prove Fer­mat’s Last The­o­rem.

Cross-do­main consequentialism

A marginally more de­tailed and hence the­ory-laden view of gen­eral in­tel­li­gence, from the stand­point of ad­vanced agent prop­er­ties, is that we can see gen­eral in­tel­li­gence as “gen­eral cross-do­main learn­ing and con­se­quen­tial­ism”.

That is, we can (ar­guendo) view gen­eral in­tel­li­gence as: the abil­ity to learn to model a wide va­ri­ety of do­mains, and to con­struct plans that op­er­ate within and across those do­mains.

For ex­am­ple: AlphaGo can be seen as try­ing to achieve the con­se­quence of a win­ning Go po­si­tion on the game board—to steer the fu­ture into the re­gion of out­comes that AlphaGo defines as a preferred po­si­tion. How­ever, AlphaGo only plans within the do­main of le­gal Go moves, and it can’t learn any do­mains other than that. So AlphaGo can’t, e.g., make a prank phone call at night to Lee Se-Dol to make him less well-rested the next day, even though this would also tend to steer the fu­ture of the board into a win­ning state, be­cause AlphaGo wasn’t pre­pro­grammed with any tac­tics or mod­els hav­ing to do with phone calls or hu­man psy­chol­ogy, and AlphaGo isn’t a gen­eral AI that could learn those new do­mains.

On the other hand, if a gen­eral AI were given the task of caus­ing a cer­tain Go board to end up in an out­come defined as a win, and that AI had ‘sig­nifi­cantly more gen­er­ally ap­pli­ca­ble than chim­panzee in­tel­li­gence’ on a suffi­cient level, that Ar­tifi­cial Gen­eral In­tel­li­gence might learn what hu­mans are, learn that there’s a hu­man try­ing to defeat it on the other side of the Go board, re­al­ize that it might be able to win the Go game more effec­tively if it could make the hu­man play less well, re­al­ize that to make the hu­man play less well it needs to learn more about hu­mans, learn about hu­mans need­ing sleep and sleep be­com­ing less good when in­ter­rupted, learn about hu­mans wak­ing up to an­swer phone calls, learn how phones work, learn that some In­ter­net ser­vices con­nect to phones…

If we con­sider an ac­tual game of Go, rather than a log­i­cal game of Go, then the state of the Go board at the end of the game is pro­duced by an enor­mous and tan­gled causal pro­cess that in­cludes not just the prox­i­mal moves, but the AI al­gorithm that chooses the moves, the cluster the AI is run­ning on, the hu­mans who pro­grammed the cluster; and also, on the other side of the board, the hu­man mak­ing the moves, the pro­fes­sional pride and fi­nan­cial prizes mo­ti­vat­ing the hu­man, the car that drove the hu­man to the game, the amount of sleep the hu­man got that night, all the things all over the world that didn’t in­ter­rupt the hu­man’s sleep but could have, and so on. There’s an enor­mous lat­tice of causes that lead up to the AI’s and the hu­man’s ac­tual Go moves.

We can see the cog­ni­tive job of an agent in gen­eral as “se­lect poli­cies or ac­tions which lead to a more preferred out­come”. The enor­mous lat­tice of real-world causes lead­ing up to the real-world Go game’s fi­nal po­si­tion, means that an enor­mous set of pos­si­ble in­ter­ven­tions could po­ten­tially steer the real-world fu­ture into the re­gion of out­comes where the AI won the Go game. But these causes are go­ing through all sorts of differ­ent do­mains on their way to the fi­nal out­come, and cor­rectly choos­ing from the much wider space of in­ter­ven­tions means you need to un­der­stand all the do­mains along the way. If you don’t un­der­stand hu­mans, un­der­stand­ing phones doesn’t help; the prank phone call event goes through the sleep de­pri­va­tion event, and to cor­rectly model events hav­ing to do with sleep de­pri­va­tion re­quires know­ing about hu­mans.

Deep com­mon­al­ities across cog­ni­tive domains

To the ex­tent one cred­its the ex­is­tence of ‘sig­nifi­cantly more gen­eral than chim­panzee in­tel­li­gence’, it im­plies that there are com­mon cog­ni­tive sub­prob­lems of the huge va­ri­ety of prob­lems that hu­mans can (learn to) solve, de­spite the sur­face-level differ­ences of those do­mains. Or at least, the way hu­mans solve prob­lems in those do­mains, the cog­ni­tive work we do must have deep com­mon­al­ities across those do­mains. Th­ese com­mon­al­ities may not be visi­ble on an im­me­di­ate sur­face in­spec­tion.

Imag­ine you’re an an­cient Greek who doesn’t know any­thing about the brain hav­ing a vi­sual cor­tex. From your per­spec­tive, ship cap­tains and smiths seem to be do­ing a very differ­ent kind of work; ships and anvils seem like very differ­ent ob­jects to know about; it seems like most things you know about ships don’t carry over to know­ing about anvils. Some­body who learns to fight with a spear, does not there­fore know how to fight with a sword and shield; they seem like quite differ­ent weapon sets.

(Since, by as­sump­tion, you’re an an­cient Greek, you’re prob­a­bly also not likely to won­der any­thing along the lines of “But wait, if these tasks didn’t all have at least some forms of cog­ni­tive la­bor in com­mon deep down, there’d be no rea­son for hu­mans to be si­mul­ta­neously bet­ter at all of them than other pri­mates.”)

Only af­ter learn­ing about the ex­is­tence of the cere­bral cor­tex and the cere­bel­lum and some hy­pothe­ses about what those parts of the brain are do­ing, are you likely to think any­thing along the lines of:

“Ship-cap­tain­ing and smithing and spearfight­ing and sword­fight­ing look like they all in­volve us­ing tem­po­ral hi­er­ar­chies of chun­ked tac­tics, which is a kind of thing the cor­ti­cal al­gorithm is hy­poth­e­sized to do. They all in­volve re­al­time mo­tor con­trol with er­ror cor­rec­tion, which is a kind of thing the cere­bel­lar cor­tex is hy­poth­e­sized to do. So if the hu­man cere­bral cor­tex and cere­bel­lar cor­tex are larger or run­ning bet­ter al­gorithms than chim­panzees’ cere­brums and cere­bel­lums, hu­mans be­ing bet­ter at learn­ing and perform­ing this kind of deep un­der­ly­ing cog­ni­tive la­bor that all these sur­face-differ­ent tasks have in com­mon, could ex­plain why hu­mans are si­mul­ta­neously bet­ter than chim­panzees at learn­ing and perform­ing ship­build­ing, smithing, spearfight­ing, and sword­fight­ing.”

This ex­am­ple is hugely over­sim­plified, in that there are far more differ­ences go­ing on be­tween hu­mans and chim­panzees than just larger cere­brums and cere­bel­lums. Like­wise, learn­ing to build ships in­volves de­liber­ate prac­tice which in­volves main­tain­ing mo­ti­va­tion over long chains of vi­su­al­iza­tion, and many other cog­ni­tive sub­prob­lems. Fo­cus­ing on just two fac­tors of ‘deep’ cog­ni­tive la­bor and just two mechanisms of ‘deep’ cog­ni­tive perfor­mance is meant more as a straw illus­tra­tion of what the much more com­pli­cated real story would look like.

But in gen­eral, the hy­poth­e­sis of gen­eral in­tel­li­gence seems like it should cash out as some ver­sion of: “There’s some set of new cog­ni­tive al­gorithms, plus im­prove­ments to ex­ist­ing al­gorithms, plus big­ger brains, plus other re­sources—we don’t know how many things like this there are, but there’s some set of things like that—which, when added to pre­vi­ously ex­ist­ing pri­mate and ho­minid ca­pa­bil­ities, cre­ated the abil­ity to do bet­ter on a broad set of deep cog­ni­tive sub­prob­lems held in com­mon across a very wide va­ri­ety of hu­manly-ap­proach­able sur­face-level prob­lems for learn­ing and ma­nipu­lat­ing do­mains. And that’s why hu­mans do bet­ter on a huge va­ri­ety of do­mains si­mul­ta­neously, de­spite evolu­tion hav­ing not pre­pro­grammed us with new in­stinc­tual knowl­edge or al­gorithms for all those do­mains sep­a­rately.”

Un­der­es­ti­mat­ing cog­ni­tive commonalities

The above view sug­gests a di­rec­tional bias of un­cor­rected in­tu­ition: Without an ex­plicit cor­rec­tion, we may tend to in­tu­itively un­der­es­ti­mate the similar­ity of deep cog­ni­tive la­bor across seem­ingly differ­ent sur­face prob­lems.

On the sur­face, a ship seems like a differ­ent ob­ject from a smithy, and the spear seems to in­volve differ­ent tac­tics from a sword. With our at­ten­tion go­ing to these visi­ble differ­ences, we’re un­likely to spon­ta­neously in­vent a con­cept of ‘re­al­time mo­tor con­trol with er­ror cor­rec­tion’ as a kind of ac­tivity performed by a ‘cere­bel­lum’—es­pe­cially if our civ­i­liza­tion doesn’t know any neu­ro­science. The deep cog­ni­tive la­bor in com­mon goes un­seen, not just be­cause we’re not pay­ing at­ten­tion to the in­visi­ble con­stants of hu­man in­tel­li­gence, but be­cause we don’t have the the­o­ret­i­cal un­der­stand­ing to imag­ine in any con­crete de­tail what could pos­si­bly be go­ing on.

This sug­gests an ar­gu­ment from pre­dictable up­dat­ing: if we knew even more about how gen­eral in­tel­li­gence ac­tu­ally worked in­side the hu­man brain, then we would be even bet­ter able to con­cretely vi­su­al­ize deep cog­ni­tive prob­lems shared be­tween differ­ent sur­face-level do­mains. We don’t know at pre­sent how to build an in­tel­li­gence that learns a par-hu­man va­ri­ety of do­mains, so at least some of the deep com­mon­al­ities and cor­re­spond­ing similar al­gorithms across those do­mains, must be un­known to us. Then, ar­guendo, if we bet­ter un­der­stood the true state of the uni­verse in this re­gard, our first-or­der/​un­cor­rected in­tu­itions would pre­dictably move fur­ther along the di­rec­tion that our be­lief pre­vi­ously moved when we learned about cere­bral cor­tices and cere­bel­lums. There­fore, to avoid vi­o­lat­ing prob­a­bil­ity the­ory by fore­see­ing a pre­dictable up­date, our sec­ond-or­der cor­rected be­lief should already be that there is more in com­mon be­tween differ­ent cog­ni­tive tasks than we in­tu­itively see how to com­pute.


In sum this sug­gests a defla­tion­ary psy­cholog­i­cal ac­count of a di­rec­tional bias of un­cor­rected in­tu­itions to­ward gen­eral-in­tel­li­gence skep­ti­cism: Peo­ple in­vent the­o­ries of dis­tinct in­tel­li­gences and nonover­lap­ping spe­cial­iza­tions, be­cause (a) they are look­ing to­ward so­cially salient hu­man-hu­man differ­ences in­stead of hu­man-vs-chim­panzee differ­ences, (b) they have failed to cor­rect for the fad­ing of in­visi­ble con­stants such as hu­man in­tel­li­gence, and (c) they have failed to ap­ply an ex­plicit cor­rec­tion for the ex­tent to which we feel like we un­der­stand sur­face-level differ­ences but are ig­no­rant of the cog­ni­tive com­mon­al­ities sug­gested by the gen­eral hu­man perfor­mance fac­tor.

(The usual cau­tions about psy­chol­o­giz­ing ap­ply: you can’t ac­tu­ally get em­piri­cal data about the real world by ar­gu­ing about peo­ple’s psy­chol­ogy.) <div>

Nat­u­rally cor­re­lated AI capabilities

Few peo­ple in the field would out­right dis­agree with ei­ther the state­ment “hu­mans have sig­nifi­cantly more widely ap­pli­ca­ble cog­ni­tive abil­ities than other pri­mates” or, or the other side, “no mat­ter how in­tel­li­gent you are, if your brain fits in­side the phys­i­cal uni­verse, you might not be able to re­verse SHA-4096 hashes”. But even tak­ing both those state­ments for granted, there seems to be a set of policy-rele­vant fac­tual ques­tions about, roughly, to what de­gree gen­eral in­tel­li­gence is likely to shorten the prag­matic dis­tance be­tween differ­ent AI ca­pa­bil­ities.

For ex­am­ple, con­sider the fol­low­ing (straw) amaz­ing sim­ple solu­tion to all of AI al­ign­ment:

“Let’s just de­velop an AI that knows how to do good things but not bad things! That way, even if some­thing goes wrong, it won’t know how to hurt us!”

To which we re­ply: “That’s like ask­ing for an AI that un­der­stands how to drive blue cars but not red cars. The cog­ni­tive work you need to do in or­der to drive a blue car is very similar to the cog­ni­tive la­bor re­quired to drive a red car; an agent that can drive a blue car is only a tiny step away from driv­ing a red car. In fact, you’d pretty much have to add de­sign fea­tures speci­fi­cally in­tended to pre­vent the agent from un­der­stand­ing how to drive a car if it’s painted red, and if some­thing goes wrong with those fea­tures, you’ll have a red-car-driv­ing-ca­pa­ble agent on your hands.”

“I don’t be­lieve in this so-called gen­eral-car-driv­ing-in­tel­li­gence,” comes the re­ply. “I see no rea­son why abil­ity at driv­ing blue cars has to be so strongly cor­re­lated with driv­ing red cars; they look pretty differ­ent to me. Even if there’s a kind of agent that’s good at driv­ing both blue cars and red cars, it’d prob­a­bly be pretty in­effi­cient com­pared to a spe­cial­ized blue-car-driv­ing or red-car-driv­ing in­tel­li­gence. Any­one who was con­struct­ing a car-driv­ing al­gorithm that only needed to work with blue cars, would not nat­u­rally tend to pro­duce an al­gorithm that also worked on red cars.”

“Well,” we say, “maybe blue cars and red cars look differ­ent. But if you did have a more con­crete and cor­rect idea about what goes on in­side a robotic car, and what sort of com­pu­ta­tions it does, you’d see that the com­pu­ta­tional sub­prob­lems of driv­ing a blue car are pretty much iden­ti­cal to the com­pu­ta­tional sub­prob­lems of driv­ing a red car.”

“But they’re not ac­tu­ally iden­ti­cal,” comes the re­ply. “The set of red cars isn’t ac­tu­ally iden­ti­cal to the set of blue cars and you won’t ac­tu­ally en­counter ex­actly iden­ti­cal prob­lems in driv­ing these non-over­lap­ping sets of phys­i­cal cars go­ing to differ­ent places.”

“Okay,” we re­ply, “that’s ad­mit­tedly true. But in or­der to re­li­ably drive any blue car you might get handed, you need to be able to solve an ab­stract vol­ume of not-pre­cisely-known-in-ad­vance cog­ni­tive sub­prob­lems. You need to be able to drive on the road re­gard­less of the ex­act ar­range­ment of the as­phalt. And that’s the same range of sub­prob­lems re­quired to drive a red car.”

We are, in this case, talk­ing to some­one who doesn’t be­lieve in color-gen­eral car-driv­ing in­tel­li­gence or that color-gen­eral car-driv­ing is a good or nat­u­ral way to solve car-driv­ing prob­lems. In this par­tic­u­lar case it’s an ob­vi­ous straw po­si­tion be­cause we’ve picked two tasks that are ex­tremely similar in an in­tu­itively ob­vi­ous way; a hu­man trained to drive blue cars does not need any sep­a­rate prac­tice at all to drive red cars.

For a straw po­si­tion at the op­po­site ex­treme, con­sider: “I just don’t be­lieve you can solve log­i­cal Tic-Tac-Toe with­out some deep al­gorithm that’s gen­eral enough to do any­thing a hu­man can. There’s no safe way to get an AI that can play Tic-Tac-Toe with­out do­ing things dan­ger­ous enough to re­quire solv­ing all of AI al­ign­ment. Be­ware the cog­ni­tive bi­ases that lead you to un­der­es­ti­mate how much deep cog­ni­tive la­bor is held in com­mon be­tween tasks that merely ap­pear differ­ent on the sur­face!”

To which we re­ply, “Con­trary to some se­ri­ous pre­dic­tions, it turned out to be pos­si­ble to play su­per­hu­man Go with­out gen­eral AI, never mind Tic-Tac-Toe. Some­times there re­ally are spe­cial­ized ways of do­ing things, the end.”

Between these two ex­tremes lie more plau­si­ble po­si­tions that have been se­ri­ously held and de­bated, in­clud­ing:

  • The prob­lem of mak­ing good pre­dic­tions re­quires a sig­nifi­cantly smaller sub­set of the abil­ities and strate­gies used by a gen­eral agent; an Or­a­cle won’t be easy to im­me­di­ately con­vert to an agent.

  • An AI that only gen­er­ates plans for hu­mans to im­ple­ment, solves less dan­ger­ous prob­lems than a gen­eral agent, and is not an im­me­di­ate neigh­bor of a very dan­ger­ous gen­eral agent.

  • If we only try to make su­per­hu­man AIs meant to as­sist but not re­place hu­mans, AIs de­signed to op­er­ate only with hu­mans in the loop, the same tech­nol­ogy will not im­me­di­ately ex­tend to build­ing au­tonomous su­per­in­tel­li­gences.

  • It’s pos­si­ble to have an AI that is, at a given mo­ment, a su­per­hu­manly good en­g­ineer but not very good at mod­el­ing hu­man psy­chol­ogy; an AI with do­main knowl­edge of ma­te­rial en­g­ineer­ing does not have to be already in im­me­di­ate pos­ses­sion of all the key knowl­edge for hu­man psy­chol­ogy.

Ar­guably, these fac­tual ques­tions have in com­mon that they re­volve about the dis­tance be­tween differ­ent cog­ni­tive do­mains—given a nat­u­ral de­sign for an agent that can do X, how close is it in de­sign space to an agent that can do Y? Is it ‘driv­ing blue cars vs. driv­ing red cars’ or ‘Tic-Tac-Toe vs. clas­sify­ing pic­tures of cats’?

(Re­lated ques­tions arise in any safety-re­lated pro­posal to di­vide an AI’s in­ter­nal com­pe­ten­cies into in­ter­nal do­mains, e.g. for pur­poses of min­i­miz­ing the num­ber of in­ter­nal goals with the power to re­cruit sub­goals across any known do­main.)

It seems like in prac­tice, differ­ent be­liefs about ‘gen­eral in­tel­li­gence’ may ac­count for a lot of the dis­agree­ment about “Can we have an AI that X-es with­out that AI be­ing 30 sec­onds away from be­ing ca­pa­ble of Y-ing?” In par­tic­u­lar, differ­ent be­liefs about:

  • To what de­gree most in­ter­est­ing/​rele­vant do­main prob­lems, de­com­pose well into a similar class of deep cog­ni­tive sub­prob­lems;

  • To what de­gree whack­ing on an in­ter­est­ing/​rele­vant prob­lem with gen­eral in­tel­li­gence is a good or nat­u­ral way to solve it, com­pared to de­vel­op­ing spe­cial­ized al­gorithms (that can’t just be de­vel­oped by a gen­eral in­tel­li­gence (with­out that AGI pay­ing prag­mat­i­cally very-difficult-to-pay costs in com­pu­ta­tion or sam­ple com­plex­ity)).

To the ex­tent that you as­sign gen­eral in­tel­li­gence a more cen­tral role, you may tend in gen­eral to think that com­pe­tence in do­main X is likely to be nearer to com­pe­tence at do­main Y. (Although not to an un­limited de­gree, e.g. wit­ness Tic-Tac-Toe or re­vers­ing a SHA-4096 hash.)

Re­la­tion to ca­pa­bil­ity gain theses

How much credit one gives to ‘gen­eral in­tel­li­gence’ is not the same ques­tion as how much credit one gives to is­sues of rapid ca­pa­bil­ity gains, su­per­in­tel­li­gence, and the pos­si­ble in­ter­me­di­ate event of an in­tel­li­gence ex­plo­sion. The ideas can definitely be pried apart con­cep­tu­ally:

  • An AI might be far more ca­pa­ble than hu­mans in virtue of run­ning or­ders of mag­ni­tude faster, and be­ing able to ex­pand across mul­ti­ple clusters shar­ing in­for­ma­tion with much higher band­width than hu­man speech, rather than the AI’s gen­eral in­tel­li­gence be­ing al­gorith­mi­cally su­pe­rior to hu­man gen­eral in­tel­li­gence in a deep sense noteE.g. in the sense of hav­ing lower sam­ple com­plex­ity and hence be­ing able to de­rive cor­rect an­swers us­ing fewer ob­ser­va­tions than hu­mans try­ing to do the same over rel­a­tively short pe­ri­ods of time. or an in­tel­li­gence ex­plo­sion of al­gorith­mic self-im­prove­ment hav­ing oc­curred.

  • If it’s cheaper for an AI with high lev­els of spe­cial­ized pro­gram­ming abil­ity to ac­quire other new spe­cial­ized ca­pa­bil­ities than for a hu­man to do the same—not be­cause of any deep al­gorithm of gen­eral in­tel­li­gence, but be­cause e.g. hu­man brains can’t evolve new cor­ti­cal ar­eas over the rele­vant times­pan—then this could lead to an ex­plo­sion of other cog­ni­tive abil­ities ris­ing to su­per­hu­man lev­els, with­out it be­ing in gen­eral true that there were deep similar sub­prob­lems be­ing solved by similar deep al­gorithms.

In prac­tice, it seems to be an ob­served fact that peo­ple who give more credit to the no­tion of gen­eral in­tel­li­gence ex­pect higher re­turns on cog­ni­tive rein­vest­ment, and vice versa. This cor­re­la­tion makes sense, since:

  • The more differ­ent sur­face do­mains share un­der­ly­ing sub­prob­lems, the higher the re­turns on cog­ni­tive in­vest­ment in get­ting bet­ter at those deep sub­prob­lems.

  • The more you think an AI can im­prove its in­ter­nal al­gorithms in faster or deeper ways than hu­man neu­rons up­dat­ing, the more this ca­pa­bil­ity is it­self a kind of Gen­eral Abil­ity that would lead to ac­quiring many other spe­cial­ized ca­pa­bil­ities faster than hu­man brains would ac­quire them. noteIt seems con­cep­tu­ally pos­si­ble to be­lieve, though this be­lief has not been ob­served in the wild, that self-pro­gram­ming minds have some­thing wor­thy of be­ing called ‘gen­eral in­tel­li­gence’ but that hu­man brains don’t.

It also seems to make sense for peo­ple who give more credit to gen­eral in­tel­li­gence, be­ing more con­cerned about ca­pa­bil­ity-gain-re­lated prob­lems in gen­eral; they are more likely to think that an AI with high lev­els of one abil­ity is likely to be able to ac­quire an­other abil­ity rel­a­tively quickly (or im­me­di­ately) and with­out spe­cific pro­gram­mer efforts to make that hap­pen.


  • Advanced agent properties

    How smart does a ma­chine in­tel­li­gence need to be, for its nice­ness to be­come an is­sue? “Ad­vanced” is a broad term to cover cog­ni­tive abil­ities such that we’d need to start con­sid­er­ing AI al­ign­ment.