Probability rules
To summarize what we have seen, we have talked about what joint, conditional, and marginal probabilities are. Here is a quick review of how these concepts relate to each other. Define two generic events A and B:
1) Conditional probabilities:
\[p(A|B)=\frac{p(A,B)}{p(B)}\] \[p(B|A)=\frac{p(A,B)}{p(A)}\]
2) Due to (1), the joint probability factorizes in conditional times marginal probability:
\[p(A,B)=p(A|B)\times p(B)\] \[p(A,B)=p(B|A)\times p(A)\]
3) Two events A and B are independent if the probability of one event does not change given that we know the other event has occurred
\[p(A|B)=p(A)\] \[p(B|A)=p(B)\]
Therefore, independence between two events imply that we can just multiply the individual marginal probabilities to get the joint probability:
\[p(A,B)=p(A|B) \times p(B) = p(A)\times p(B)\]
4) Probabilities are constrained to be non-negative and to be smaller or equal to 1
\[0 \leq p(A) \leq 1\]
5) For mutually exclusive events, the probability of all these events occurring but A is given by one minus the probability of event A occurring
\[p(A^c )=1-p(A)\]
where \(A^c\) is the complement of A (all other events but A).
6) For mutually exclusive events, the marginal probability can be calculated as:
\[p(A)=\sum_{j=1}^{J} p(A,B=j) = \sum_{j=1}^{J} p(A|B=j)p(B=j)\]
where j refers to a particular outcome that event B can take. This is called the law of total probability. For example, when talking about death penalty and race, we said that:
\[p(D=Yes)=p(D=Yes,R=Black)+p(D=Yes,R=White)\]
Alternatively, we can also state that: \[p(D=Yes)=p(D=Yes|R=Black)p(R=Black)+p(D=Yes|R=White)p(R=White)\]
We can generalize the law of total probability for the situation when the outcomes for event B are continuous by substituting summation by integration.
7) Based on the results above, Bayes theorem is given by:
\[p(A|B)=\frac{p(A,B)}{p(B)}=\frac{p(B|A)p(A)}{p(B)}=\frac{p(B|A)p(A)}{\sum_{j=1}^{J} p(B,A=j)}=\frac{p(B|A)p(A)}{\sum_{j=1}^{J} p(B|A=j)p(A=j)}\]
A note on composite events
It is important to realize that we can also have composite events. For instance, say that event X is comprised of events B and C. In other words, \(p(X)=p(B,C)\). Then:
\[p(A,X)=p(A|X)p(X)\] \[=p(A|B,C)p(B,C)\] \[=p(A|B,C)p(B|C)p(C)\]
This type of decomposition is often very useful when there is a natural sequence of events. For instance, an earthquake can happen (event C), which may lead to structural problems in buildings (event B), which may then lead to injuries and mortality (event A).
You can read more on these probability rules in pages 1-7 in (Downey 2012).
Comments?
Send me an email at