Inductive prior

An “in­duc­tive prior” is a state of be­lief, be­fore see­ing any ev­i­dence, which is con­ducive to learn­ing when the ev­i­dence fi­nally ap­pears. A clas­sic ex­am­ple would be ob­serv­ing a coin come up heads or tails many times. If the coin is bi­ased to come up heads 14 of the time, the in­duc­tive prior from Laplace’s Rule of Suc­ces­sion will start pre­dict­ing fu­ture flips to come up tails with 34 prob­a­bil­ity. The max­i­mum en­tropy prior for the coin, which says that ev­ery coin­flip has a 50% chance of com­ing up heads and that all se­quences of heads and tails are equally prob­a­ble, will never start to pre­dict that the next flip will be heads, even af­ter ob­serv­ing the coin come up heads thirty times in a row.

The prior in Solomonoff in­duc­tion is an­other ex­am­ple of an in­duc­tive prior—far more pow­er­ful, far more com­pli­cated, and en­tirely unim­ple­mentable on phys­i­cally pos­si­ble hard­ware.

Children:

  • Solomonoff induction

    A sim­ple way to su­per­in­tel­li­gently pre­dict se­quences of data, given un­limited com­put­ing power.

  • Laplace's Rule of Succession

    Sup­pose you flip a coin with an un­known bias 30 times, and see 4 heads and 26 tails. The Rule of Suc­ces­sion says the next flip has a 532 chance of show­ing heads.

  • Universal prior

    A “uni­ver­sal prior” is a prob­a­bil­ity dis­tri­bu­tion con­tain­ing all the hy­pothe­ses, for some rea­son­able mean­ing of “all”. E.g., “ev­ery pos­si­ble com­puter pro­gram that com­putes prob­a­bil­ities”.

Parents:

  • Ignorance prior

    Key equa­tions for quan­ti­ta­tive Bayesian prob­lems, de­scribing ex­actly the right shape for what we be­lieved be­fore ob­ser­va­tion.