## The Binomial Distribution

Permutations and Combinations: When we flip a coin twice, we observe a particular sequence or permutation; for example, a head followed by a tail. In contrast, a head and a tail, in any order, is called a combination. A head and a tail can exist as one of two possible permutations xht = (xh, xt), or xth = (xt, xh). The probability of observing each of these permutations is the same because the order in which heads and tails occurs does not affect their probabilities. Thus,

 p(xht | θ) = $p\left({x}_{h}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)×p\left({x}_{t}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)$ (E.1) = θ × (1 − θ) (E.2) p(xth | θ) = $p\left({x}_{t}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)×p\left({x}_{h}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)$ (E.3) = $\left(1-\theta \right)×\theta ,$ (E.4)

and if we assume θ = 0.6 then the probability of the permutations p(xht|θ) = p(xth|θ) = 0.24. As each of these two permutations occurs with probability 0.24, it follows that the probability of observing a head and a tail in any order, or equivalently, the probability of a combination that contains a head and a tail, is

 $\begin{array}{ccc}p\left({\mathbf{x}}_{\mathit{ht}}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)+p\left({\mathbf{x}}_{\mathit{th}}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)& =& 0.48.\end{array}$ (E.5)

We can clean up the notation by defining x as a specific combination, such as x = {xt, xh}, where this is read as one head out of two coin flips in any order, and where we use curly brackets to represent a combination. Thus, we can write

 p(x | θ) = p({xt, xh | θ}) (E.6) = 2 × θ × (1 − θ) (E.7) = 0.48, (E.8)

where θ(1 – θ) is the probability that any permutation contains exactly one head and one tail, and 2 is the number of permutations that contain exactly one head and one tail.

We can reassure ourselves that Equation E.8 is true by working out the probability of observing the three possible combinations that can result from two coin flips, namely, {xt, xt}, {xt, xh}, and {xh, xh}, given a specific value for θ. The combination {xt, xt} includes only one permutation (xt, xt), so

 $\begin{array}{ccc}p\left(\left\{{x}_{t},{x}_{t}\right\}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)& =& p\left({x}_{t}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)×p\left({x}_{t}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)=0.16.\end{array}$ (E.9)

Similarly, for the combination {xh, xh}, we have

 $\begin{array}{ccc}p\left(\left\{{x}_{h},{x}_{h}\right\}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)& =& p\left({x}_{h}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)×p\left({x}_{h}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)=0.36.\end{array}$ (E.10)

So, given that the only possible three combinations from two coin flips contain zero heads {xt, xt}, one head {xh, xt}, or two heads {xh, xh}, with probabilities

 $\begin{array}{ccc}p\left(\left\{{x}_{t},{x}_{t}\right\}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)& =& 0.16\end{array}$ (E.11) $\begin{array}{ccc}p\left(\left\{{x}_{h},{x}_{h}\right\}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)& =& 0.36\end{array}$ (E.12) $\begin{array}{ccc}p\left(\left\{{x}_{h},{x}_{t}\right\}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)& =& 0.48,\end{array}$ (E.13)

the sum of these probabilities is one

 $0.16+0.36+0.48=1.$ (E.14)

The general lesson to be drawn from this simple example is as follows. To find the probability of observing a combination that contains n heads and Nn tails, first find the probability of one such permutation, and multiply this by the number of permutations that contain n heads and Nn tails, where this number is a binomial coefficient (described next).

The Binomial Coefficient: The number of permutations that contain exactly x heads amongst N coin flips is

 $\begin{array}{ccc}{C}_{N,x}& =& N!/\left(x\left(N-x\right)!\right),\end{array}$ (E.15)

where CN,x is a binomial coefficient and is pronounced N choose x. If a coin is flipped N = 10 times and the number of heads is x = 7 then the number of permutations that contain x = 7 heads and Nx = 3 tails is CN,x = 120. If a coin has a bias θ = 0.7 then the probability of obtaining any one of these permutations, such as, x7 = (xh, xh, xh, xh, xh, xh, xh, xt, xt, xt), is

 $\begin{array}{c}p\left({\mathbf{x}}_{7}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)\end{array}={\theta }^{7}{\left(1-\theta \right)}^{3}=2.223×{10}^{-3}.$ (E.16)

Given that the probability of obtaining any other permutation with x = 7 heads and Nx tails is also 2.223 × 10−3, and that there are CN,x such permutations, it follows that the probability of obtaining x heads and Nx tails in any order (ie a combination) is

 p(x | θ) = ${C}_{N,x}\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}p\left({\mathrm{x}}_{7}|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta \right)$ (E.17) = $120×\left(2.223×{10}^{-3}\right)$ (E.18) = 0.267. (E.19)

The Binomial Distribution: If we flip this coin N times then the probability of observing x heads given a coin bias θ is

 $\begin{array}{ccc}p\left(x|\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}\theta ,N\right)& =& {C}_{N,x}\phantom{\rule[-0.0ex]{0.2em}{0.0ex}}{\theta }^{x}{\left(1-\theta \right)}^{N-x}.\end{array}$ (E.20)

Figure E.1.: The binomial distribution. Given a coin with a bias of θ = 0.7, which is flipped N = 10 times, the probability p(x|θ) of obtaining different numbers x of heads defines a binomial distribution.

The binomial distribution for N = 10 and θ = 0.7 is shown in Figure E.1. (see Section 4.1, p74).

The binomial distribution is a cornerstone in the analysis of binary events. As the number N (eg coin flips) increases, given some mild assumptions, the binomial distribution becomes increasingly like the Gaussian distribution (see Chapter 5 and Appendix F). Because the Gaussian distribution is mathematically convenient, this approximation allows the binomial distribution to be replaced with the Gaussian distribution in a wide variety of contexts.