The term “bit” is over­loaded. It can mean any of the fol­low­ing:

  1. An el­e­ment of the set \(\mathbb B\), which has two el­e­ments. Th­ese el­e­ments are some­times called 0 and 1, or true and false, or yes and no. See Bit (ab­stract).

  2. A unit of data, namely, the amount of data re­quired to sin­gle out one mes­sage from a set of two, or, equiv­a­lently, the amount of data re­quired to cut a set of pos­si­ble mes­sages in half. See Bit of data.

  3. A unit of (sub­jec­tive) in­for­ma­tion, namely, the differ­ence in cer­tainty be­tween hav­ing no idea which way a coin is go­ing to come up, and be­ing en­tirely cer­tain that it’s go­ing to come up heads. While this unit of in­for­ma­tion is col­lo­quially known as a “bit” (for his­tor­i­cal rea­sons), it is more prop­erly known as a Shan­non.

  4. A unit of ev­i­dence, namely, a \(2 : 1\) rel­a­tive like­li­hood in fa­vor of one out­come over an­other. See Bit of ev­i­dence and/​or Bayes’ rule: Log-odds form.

The com­mon theme across all the uses listed above is the num­ber 2. An ab­stract bit is one of two val­ues. A bit of data is a por­tion of a rep­re­sen­ta­tion of a mes­sage that cuts the set of pos­si­ble mes­sages in half (i.e., the \(\log_2\) of the num­ber of pos­si­ble mes­sages). A bit of in­for­ma­tion (aka a Shan­non) is the amount of in­for­ma­tion in the an­swer to a yes-or-no ques­tion about which the ob­server was max­i­mally un­cer­tain (i.e., the \(\log_2\) of a prob­a­bil­ity). A bit of ev­i­dence in fa­vor of A over B is an ob­ser­va­tion which pro­vides twice as much sup­port for A as it does for B (i.e., the \(\log_2\) of a rel­a­tive like­li­hood). Thus, if you see some­one us­ing bits as units (a bit of data, or a bit of ev­i­dence, etc.) you can bet that they took a \(\log_2\) of some­thing some­where along the way.

Un­for­tu­nately, ab­stract bits break this pat­tern, so if you see some­one talk­ing about “bits” with­out dis­am­biguat­ing what sort of bit they mean, the most you can be sure of is that they’re talk­ing about some­thing that has to do with the num­ber 2. Un­less they’re just us­ing the word “bit” to mean “small piece,” in which case you’re in a bit of trou­ble.



  • Mathematics

    Math­e­mat­ics is the study of num­bers and other ideal ob­jects that can be de­scribed by ax­ioms.