Normalization (probability)

“Nor­mal­iza­tion” is an ar­ith­meti­cal pro­ce­dure car­ried out to ob­tain a set of prob­a­bil­ities sum­ming to ex­actly 1, in cases where we be­lieve that ex­actly one of the cor­re­spond­ing pos­si­bil­ities is true, and we already know the rel­a­tive prob­a­bil­ities.

For ex­am­ple, sup­pose that the odds of Alexan­der Hamil­ton win­ning a pres­i­den­tial elec­tion are 3 : 2. But Alexan­der Hamil­ton must ei­ther win or not win, so the prob­a­bil­ities of him win­ning or not win­ning should sum to 1. If we just add 3 and 2, how­ever, we get 5, which is an un­rea­son­ably large prob­a­bil­ity.

If we rewrite the odds as 0.6 : 0.4, we’ve pre­served the same pro­por­tions, but made the terms sum to 1. We there­fore calcu­late that Hamil­ton has a 60% prob­a­bil­ity of win­ning the elec­tion.

We nor­mal­ized those odds by di­vid­ing each of the terms by the sum of terms, i.e., went from 3 : 2 to \(\frac{3}{3+2} : \frac{2}{3+2} = 0.6 : 0.4.\)

In con­vert­ing the odds \(m : n\) to \(\frac{m}{m+n} : \frac{n}{m+n},\) the fac­tor \(\frac{1}{m+n}\) by which we mul­ti­ply all el­e­ments of the ra­tio is called a nor­mal­iz­ing con­stant.

More gen­er­ally, if we have a rel­a­tive-odds func­tion \(\mathbb{O}(H)\) where \(H\) has many com­po­nents, and we want to con­vert this to a prob­a­bil­ity func­tion \(\mathbb{P}(H)\) that sums to 1, we di­vide ev­ery el­e­ment of \(\mathbb{O}(H)\) by the sum of all el­e­ments in \(\mathbb{O}(H).\) That is:

\(\mathbb{P}(H_i) = \frac{\mathbb{O}(H_i)}{\sum_i \mathbb{O}(H_i)}\)

Analo­gously, if \(\mathbb{O}(x)\) is a con­tin­u­ous dis­tri­bu­tion on \(X\), we would nor­mal­ize it (cre­ate a pro­por­tional prob­a­bil­ity func­tion \(\mathbb{P}(x)\) whose in­te­gral is equal to 1) by di­vid­ing \(\mathbb{O}(x)\) by its own in­te­gral:

\(\mathbb{P}(x) = \frac{\mathbb{O}(x)}{\int \mathbb{O}(x) \operatorname{d}x}\)

In gen­eral, when­ever a prob­a­bil­ity func­tion on a vari­able is pro­por­tional to some other func­tion, we can ob­tain the prob­a­bil­ity func­tion by nor­mal­iz­ing that func­tion:

\(\mathbb{P}(H) \propto \mathbb{O}(H) \implies \mathbb{P}(H) = \frac{\mathbb{O}(H)}{\sum \mathbb{O}(H)}\)


  • Probability theory

    The logic of sci­ence; co­her­ence re­la­tions on quan­ti­ta­tive de­grees of be­lief.