# Vinge's Principle

Vinge’s Prin­ci­ple says that, in do­mains com­pli­cated enough that perfect play is not pos­si­ble, less in­tel­li­gent agents will not be able to pre­dict the ex­act moves made by more in­tel­li­gent agents.

For ex­am­ple, if you knew ex­actly where Deep Blue would play on a chess­board, you’d be able to play chess at least as well as Deep Blue by mak­ing what­ever moves you pre­dicted Deep Blue would make. So if you want to write an al­gorithm that plays su­per­hu­man chess, you nec­es­sar­ily sac­ri­fice your own abil­ity to (with­out ma­chine aid) pre­dict the al­gorithm’s ex­act chess moves.

This is true even though, as we be­come more con­fi­dent of a chess al­gorithm’s power, we be­come more con­fi­dent that it will even­tu­ally win the chess game. We be­come more sure of the game’s fi­nal out­come, even as we be­come less sure of the chess al­gorithm’s next move. This is Vingean un­cer­tainty.

Now con­sider agents that build other agents (or build their own suc­ces­sors, or mod­ify their own code). Vinge’s Prin­ci­ple im­plies that the choice to ap­prove the suc­ces­sor agent’s de­sign must be made with­out know­ing the suc­ces­sor’s ex­act sen­sory in­for­ma­tion, ex­act in­ter­nal state, or ex­act mo­tor out­puts. In the the­ory of tiling agents, this ap­pears as the prin­ci­ple that the suc­ces­sor’s sen­sory in­for­ma­tion, cog­ni­tive state, and ac­tion out­puts should only ap­pear in­side quan­tifiers. This is Vingean re­flec­tion.

For the rule about fic­tional char­ac­ters not be­ing smarter than the au­thor, see Vinge’s Law.

Parents:

• Vingean reflection

The prob­lem of think­ing about your fu­ture self when it’s smarter than you.

• Tech­ni­cally, couldn’t we run by hand on a piece of pa­per all the com­pu­ta­tions that Deep Blue goes through, and this way “pre­dict the al­gorithm’s ex­act chess moves”? In a way in­tu­itively I feel like it’s wrong to say that Deep Blue is “bet­ter than” us at play­ing chess, or AlphaGo is “bet­ter than” us at play­ing go. I feel like it de­pends on how we define “bet­ter”, or in gen­eral “in­tel­li­gence” and/​or “skill” – if it is re­lated to a no­tion of effi­ciency vs to one of speed. Be­cause in terms of pure “com­pe­tency”, it seems like what­ever a com­puter can do, we can do it too, al­though much slower – by just ex­e­cut­ing each line one step at a time.

As far as I can tell, cur­rent AI sys­tems can just ex­plore the search space of pos­si­ble moves faster than us. They aren’t nec­es­sar­ily as effi­cient as us – ar­guably AI sys­tems are still very sam­ple-in­effi­cient (i.e. AlphaGo trained on many more games than any hu­man would be able to play in his life­time).

Clearly though run­ning through all the com­pu­ta­tions by hand would take an un­fea­si­ble amount of time. Not sure if this is just a minor philo­soph­i­cal point or an ac­tual thing one should care about. I’m still learn­ing more about the field, wouldn’t be sur­prised if some­one already talked about this differ­ence be­tween speed and effi­ciency in defin­ing in­tel­li­gence but I just haven’t found it yet.

I guess what I’m try­ing to say is that I agree with the premise that “less in­tel­li­gent agents will not be able to pre­dict the ex­act moves made by more in­tel­li­gent agents”, but I’m some­how not con­vinced that Deep­Blue or AlphaGo are “more in­tel­li­gent” than us – de­pend­ing on the defi­ni­tion of in­tel­li­gence we use. And un­der defi­ni­tions for which they are more in­tel­li­gent than us, then I don’t agree Vinge’s Prin­ci­ple ap­plies for them un­less there are time con­straints.

[The ex­act phrase that Deep Blue is “bet­ter than us at play­ing chess” – that prompted my com­ment – is ac­tu­ally men­tioned in this page un­der the “Com­pat­i­bil­ity with Vingean un­cer­tainty” para­graph]