This lecture initiates the development of probability theory that is general enough to model cases where outcomes might have different probabilities. Our first task will be to define what is a valid way of assigning probabilities. Obviously we want to rule out things like assigning negative probabilities, or probabilities that do not add up to $1$.
In this definition, we use $2^\Omega$ to mean the power set of $\Omega$ (instead of $\mathcal{P}(\Omega)$, which may be confusing in this context).
A discrete probability space is a pair $(\Omega,p)$ where $\Omega$ is a set (called the sample space) and $p : \Omega \to \mathbb{R}$ is a function (called a distribution) satisfying the following requirements:
If you are following the BH textbook, note that this definition is different from the one they use in Section 1.6. Our is simpler and equivalent for all of the settings we'll care about this quarter. The definition in Rosen Section 7.2 is closer to this.
The motivation for the first part of this definition is pretty direct: For each outcome $\omega$, the function $p$ is defining the probability of the outcome. Following our prior experience with probability, we expect this to be a positive number (requirement 1). Second, we expect that all of these probabilities add up to $1$ (requirement 2).
The second part of the definition defines a function $\Pr$ that takes as input a subset of $\Omega$ and also outputs a probability. It's defined with the same intuition as in naive probability: The probability of an event is just the sum of the probabilities of outcomes it contains.
You may have noticed that we call this a discrete probability space. There is indeed a more general notion of probability space that is able to handle "continuous" spaces like choosing real numbers. We won't discuss this direction further because it gets highly technical (with the sort of technicalities we do not want to study now).
Let $\Omega = \{H,T\}$, and define $p(H) = 3/4$ and $p(T)=1/4$. This probability space models a biased coin flip. More generally for any $0 \leq q \leq 1$ we can define $p(H) = q$ and $p(T) = 1-q$.
Let $(\Omega,\Pr)$ be a discrete probability space. Then the following hold:
When $\Omega$ is finite, the proof of this theorem is an easy generalization of the counting arguments we have seen. They are still true for infinite spaces, but one has to account for taking sums over such sets (i.e. do some calculus and take limits). We won't worry about doing the latter case rigorously in this class.
Something that doesn't quite have a direct analogue from our discussions on set theory is the notion of conditional events.
Let $A$ and $B$ be events with $\Pr(B)\neq 0$. The conditional probability of $A$ given $B$ is defined to be $$\Pr(A \mid B) = \frac{\Pr(A \cap B)}{\Pr(B)}.$$
We can compute the conditional probability of $A$ given $B$ when we have some event $A$ whose probability we want to update given that we have observed $B$.
Note that for any event $A$, we have $$\Pr(A \mid A) = \frac{\Pr(A \cap A)}{\Pr(A)} = 1. $$ This definition also gives us that $\Pr(A \cap B) = \Pr(A \mid B) \cdot \Pr(B)$.
One way of looking at this is if we restrict our attention to the event $B$ and ask, what is the probability of $A$ in this space? In this case, we are asking for the intersection of the events $A$ and $B$. Then to "zoom out" back to the sample space $\Omega$, we divide by the probability of $B$.
Another is to look at the version below the definition, where it's multiplied out. If we want to compute $\Pr(A \cap B)$, the probability that two events both occur, then one clear way to do so is to find the probability that the first event occurs ($\Pr(B)$) and then find the probability that the second occurs, assuming the first occurs ($\Pr(A \mid B)$).
Consider a roll of two fair dice.
The set of outcomes when rolling two dice is $\Omega = \{1,\ldots,6\}^2 = \{(1,1), (1,2), \ldots, (6,6)\}$. We have $|\Omega| = 36$.
We can work this problem more intuitively by observing that when we condition on $B$, it means that one of its six outcomes happens with equal probability. But only one of those six has a sum of four, so the probability is $1/6$. This type of thinking will often work, but you should be very cautious about errors in intuition. The value of the definition is that we could work the problem without appealing to intuition and be confident in our answer.
The formal calculation is \begin{align*} \Pr(B \mid A) & = \frac{\Pr(A \cap B)}{\Pr(A)} \\ &= \frac{|A\cap B|/|\Omega|}{|A|/|\Omega|} \\ &= \frac{1/36}{3/36} = \frac{1}{3}, \end{align*} confirming the intuitive approach.
The prior example shows that in general $\Pr(A\mid B) \neq \Pr(B\mid A)$. They might be equal however.
Here is an example of why caution is required until you have developed a very sharp intuition about conditional probability.
Suppose we flip two fair coins.
We want $\Pr(A\mid B)$ for the first part and $\Pr(A\mid C)$ for the second. We calculate \begin{align} \Pr(A \mid B) &= \frac{\Pr(A \cap B)}{\Pr{B}} \\ &= \frac{\Pr(\{HH\})}{\Pr(B)} \\ &= \frac{1/4}{3/4} = 1/3. \end{align} Next, \begin{align} \Pr(A \mid C) &= \frac{\Pr(A \cap C)}{\Pr{C}} \\ &= \frac{\Pr(\{HH\})}{\Pr(C)} \\ &= \frac{1/4}{1/2} = 1/2. \end{align} So these conditional probabilities are not equal. If you find this result counter-intuitive, you're not alone. This happened because conditioning on $B$ leaves three possibilities open while conditioning only $C$ leaves only two.
The definition of independence is wonderfully simple.
Events $A$ and $B$ are independent if $\Pr(A \cap B) = \Pr(A) \cdot \Pr(B)$.
In our problem-solving, there will be two ways to determine whether events are independent. First, by calculation we can show that the necessary equality holds, using some pre-established model. The next example shows how this can happen. The second way is by fiat, where we look at an experiment we're modeling, and jump straight to the conclusion that some events should be independent (to the point where we would reject any model in which they weren't). The second example gives a case where we would do this.
From the definitions, it is easy to see that if $A$ and $B$ are independent, then $\Pr(A \mid B) = \frac{\Pr(A) \cdot \Pr(B)}{\Pr(B)} = \Pr(A)$, which should fit our intuition about conditional probability and independence.
Consider a roll of a two six-sided dice and the events
All the $F_i$ and $S_j$ should be independent: The second roll doesn't depend on the outcome of the first, or vice-versa. Mathematically, for any $i$ and $j$, $$\Pr(F_i \cap S_j) = \frac{1}{36} = \frac{1}{6} \cdot \frac{1}{6} = \Pr(F_i) \cdot \Pr(S_j)$$
Equally clearly, most $T_k$ are correlated with the $F_i$: You can't get a total of 11 unless the first die comes up $5$ or $6$. The one exception is $T_7$: $\Pr(T_7 \mid F_i) = \frac{1}{6}$ and likewise for $S_i$. However, obviously $T_7$ isn't independent of the combination of $F_i$ and $S_i$s: $\Pr(T_7 \mid (F_4 \cap S_3)) = 1$ whereas $\Pr(T_7 \mid (F_4 \cap S_4)) = 0$.
Consider a coin biased to show Heads with probability $3/4$, and suppose we flip it twice. What is the probability that we get two Heads? By fiat, we declare that the events $H_1,H_2$ of getting heads on the first and second toss respectively should be independent. So we have $$ \Pr(H_1 \cap H_2) = \Pr(H_1)\Pr(H_2) = \left(\frac{3}{4}\right)^2 = \frac{9}{16}. $$
The last topic in this lecture is just a bit of algebraic manipulation away from what we've already seen today in class, but has a surprisingly wide array of applications.
For two events $A$ and $B$, $$\Pr(A|B) = \frac{\Pr(B|A) \cdot \Pr(A)}{\Pr(B)}$$
We found earlier in this lecture that the definition of conditional probability gives us $\Pr(A \cap B) = \Pr(A \mid B) \cdot \Pr(B)$. Because the event $A \cap B$ is the same as the event $B \cap A$, we know $\Pr(A \cap B) = \Pr(B \mid A) \cdot \Pr(A)$. Substituting, we find $$\Pr(A \mid B) \cdot \Pr(B) = \Pr(B \mid A) \cdot \Pr(A),$$ so $$\Pr(A|B) = \frac{\Pr(B|A) \cdot \Pr(A)}{\Pr(B)}.$$
A partition of $\Omega$ is a family of pairwise disjoint events $H_1, \dots, H_m$ covering $\Omega$. That is, $\bigcup_{i=1}^m H_i = \Omega$ and $H_i \cap H_j = \emptyset$ for all $i \neq j$.
Given a partition $H_1, \dots, H_m$ of $\Omega$ and an event $A \subseteq \Omega$, $$\Pr(A) = \sum_{i=1}^m \Pr(A \cap H_i).$$
Since we have a partition $H_1, \dots, H_m$, we have $A = A \cap \Omega = A \cap \bigcup_{i = 1}^m H_i$. Then, \begin{align*} \Pr(A) &= \Pr\left( A \cap \bigcup_{i=1}^m H_i \right) \\ &= \Pr \left( \bigcup_{i=1}^m (A \cap H_i) \right) \\ &= \sum_{i=1}^m \Pr(A \cap H_i) &\text{since $H_i$ are pairwise disjoint} \\ \end{align*}
This can be rewritten as $$\Pr(A) = \sum_{i=1}^m \Pr(A \mid H_i) \cdot \Pr(H_i).$$
The Law of Total Probability generalizes the idea that if I know $\Pr(A \cap B)$ and I know $\Pr(A \cap B^c)$, then that just gives me $\Pr(A)$ when I put them together.
The Law of Total Probability also gives me that $$\Pr(B^c|A) + \Pr(B|A) = \frac{\Pr(B^c \cap A)}{\Pr(A)} + \frac{\Pr(B \cap A)}{\Pr(A)} = \frac{\Pr(B^c \cap A) + \Pr(B \cap A)}{\Pr(A)} = \frac{\Pr(A)}{\Pr(A)} = 1.$$
One interpretation of Bayes' theorem that is particularly useful is thinking of $A$ as a hypothesis (e.g. this medicine cures the disease) and $B$ as an observation (e.g. this patient who took the medicine is now better). We'd like to know whether $A$ is true, so we run a test and get observation $B$. Does that mean we should conclude that $A$ is true? How should it affect our belief in $A$? We should think that it increases the probability of $A$ being true, but by how much?
Bayes' Theorem gives us a nice structure to talk about it. It tells us that $\Pr(A|B)$ can be described as an interaction between $\Pr(A)$ (our current belief in $A$), $\Pr(B)$ (the probability of getting better), and $\Pr(B|A)$ (the probability of the patient getting better given that the medicine does cure it). Each of these values are at least easier to grapple with - we can estimate them based on data we already have. This approach to hypothesis testing is known as Bayesian statistics, and has become popular as a method for testing the strength of models, an endeavor increasingly relevant with the advent of large computers doing large predictive computations.
One canonical example application of Bayesian statistics is in disease testing. Say I am being tested for a disease that only affects $1\%$ of the population. The test has a $10\%$ false negative rate — a person with the disease will test negative $10\%$ of the time. It has a $30\%$ false positive rate — a person without the disease will test positive $30\%$ of the time. I am tested for the disease, and it returns positive. What is the probability that I have the disease?
Here, we invoke Bayes' Theorem. Define $A$ to be the event that I have the disease and $B$ to be the event that I test positive.
First, I can map the information given to the events $A$ and $B$. We know that the false negative rate is $10\%$, so $\Pr(B^c \mid A) = 0.1$. This gives us that $\Pr(B \mid A) = 0.9$. Similarly, we know that the false positive rate is $30\%$, so $\Pr(B \mid A^c) = 0.3$. This gives us that $\Pr(B^c \mid A^c) = 0.70$.
Finally, we know that $\Pr(A) = 0.01$ because only $1\%$ of the population is affected.
For Bayes' Theorem, we need to know $\Pr(B \mid A)$, $\Pr(A)$, and $\Pr(B)$. To determine $\Pr(B)$, we need to use the Law of Total Probability: $$\Pr(B) = \Pr(A \cap B) + \Pr(A^c \cap B).$$
An application of the definition of conditional probability then tells us that $$\Pr(B) = \Pr(A) \cdot \Pr(B \mid A) + \Pr(A^c) \cdot \Pr(B \mid A^c).$$
Plugging in the values given, we note that $$\Pr(B) = 0.01 \cdot 0.9 + 0.99 \cdot 0.3 = 0.306.$$
With this, we invoke Bayes' Theorem to find that $$\Pr(A|B) = \frac{\Pr(B|A) \cdot \Pr(A)}{\Pr(B)} = \frac{0.9 \cdot 0.01}{0.306} \approx 0.029,$$ so there is actually less than a 3% chance that I even have the disease based on this test!