Recall at the beginning of the course, when we were first discussing logic, that we could define propositional statements, but at some point we hit a wall in terms of being able to express properties of objects. At that point, we had to introduce predicates into our language to be able to express such notions.
We have a very similar problem at this point, where we have been discussing and defining events explicitly whenever we want to discuss some particular set of outcomes. What we'd like to do now is develop a language that allows us to say "let $X$ be the outcome of a random die roll" or let "let $Y$ by the number of spades in a random five card poker hand". That is, we want to assign variable names to unknown random values in a rigorous and useful way, one that lets us talk about outcomes rather than just events either occurring or not.
The following definition formalizes the standard approach for doing this. At first glance, it might seem odd, or even useless. At least it is simple. The motivation will come later.
A random variable on a sample space $\Omega$ is a function $X : \Omega \to \mathbb R$.
The choice of co-domain $\mathbb R$ is more permissive than prescriptive. A lot of the time, we'll be mapping objects to a particular subset of $\mathbb R$, like $\mathbb N$ or even $\{0,1\}$. And while the standard definition of a random variable is for $\mathbb R$, this can be extended to more exotic co-domains than just $\mathbb R$, but we usually prefer numbers so that we can eventually talk about averages. In fact, for the purposes of this course, you can think of the range of the function as probabilities.
Now, you may have noticed from our discussion that random variables are not random and are not variables (a random variable is a function, as we can see from the definition above).
Having set up this mechanism, we can define the probability of a particular outcome of a random variable.
If $X$ is a random variable, we define the notation $$\Pr(X = r) = \Pr(\{\omega \in \Omega \mid X(\omega) = r\}).$$ More generally we extend this notation as follows. If $Q$ is any predicate, we define $$\Pr(Q(X)) = \Pr(\{\omega \in \Omega \mid Q(X(\omega)) \}).$$
Suppose we flip a coin three times. Let $X(\omega)$ be the random variable that equals the number of heads that appear when $\omega$ is the outcome. So we have \begin{align*} X(HHH) &= 3, \\ X(HHT) = X(HTH) = X(THH) &= 2, \\ X(TTH) = X(THT) = X(HHT) &= 1, \\ X(TTT) &= 0. \end{align*} Now, suppose that the coin is fair. Since we assume that each of the coin flips is mutually independent, this gives us a probability of $\frac 1 2 \frac 1 2 \frac 1 2 = \frac 1 8$ for any one of the above outcomes. Now, let's consider $\Pr(X = k)$:
| $k$ | $\{\omega \in \Omega \mid X(\omega) = k\}$ | $\Pr(X = k)$ |
|---|---|---|
| 3 | $\{HHH\}$ | $\frac 1 8$ |
| 2 | $\{HHT,HTH,THH\}$ | $\frac 3 8$ |
| 1 | $\{TTH,THT,HTT\}$ | $\frac 3 8$ |
| 0 | $\{TTT\}$ | $\frac 1 8$ |
Using the notation from the definition, we could write $\Pr(X \leq 1)$ to mean $\Pr(\{\omega \ : \ X(\omega) \leq 1\})$; This probability is $1/2$, since $4$ out of $8$ outcomes have at most $1$ Heads.
Continuing with the same sample space, we can define a random variable $Y$ that counts the number of tails. Then we have, for all $\omega\in\Omega$, $X(\omega) + Y(\omega)=3$.
Notation for random variables often omits the input variable. So one might define a new random variable $Z$ by saying $Z=X-Y$, or in English: $Z$ is the number of heads minus the number of tails. It is important to understand that this is just defining a new function like you did in calculus, where you'd write $h(x) = f(x)+g(x)$. For another example, we could have defined $Y$ above by $Y=3-X$.
When studying random variables and answering questions about them, a very useful concept is their distribution, which is defined next.
Let $X$ be a random variable on a sample space $\Omega$. The distribution of $X$ is the function $p_X : \mathbb{R}\to\mathbb{R}$ defined by $$ p_X(x) = \Pr(X = x). $$ This function is also called the probability mass function or pmf of $X$.
In the three-coin-toss example above, when $X$ counts the number of heads, we have $$ p_X(0) = 1/8, \quad p_X(1) = 3/8, \quad p_X(2) = 3/8, \quad p_X(3) = 1/8, $$ and $p_X(x) = 0$ for all other $x$.
Now consider $Y$, which counts the number of tails. This is a different random variable (it is a different function!). But observe that its distribution is $$ p_Y(0) = 1/8, \quad p_Y(1) = 3/8, \quad p_Y(2) = 3/8, \quad p_Y(3) = 1/8, $$ and $p_Y(x) = 0$ for all other $x$. This is exactly the same function of $p_X$. That is, $p_X = p_Y$. This is a very common situation, and the next definition records a term to describe this situation.
Let $X,Y$ be random variables. When $p_X=p_Y$, we say that $X$ and $Y$ are identically distributed.
Now consider rolling two dice, and let $X$ be the first roll and $Y$ be the second roll. We have that $X$ and $Y$ are identically distributed, and that their distributions assign $1/6$ to $x=1,2,3,4,5,6$ and $0$ to all other $x$. Now define $Z=X+Y$. What is the distribution of $Z$? You can calculate this function by hand. For example, there is only one roll {(1,1)} that gives Z = 2, but there are two rolls {(2,1), (1,2)} that give Z = 3. There are three rolls {(1,3), (2,2), (3,1)} that give Z = 4. Following similar logic, we have the following for our first few values: $$ p_Z(2) = 1/36, \quad p_Z(3) = 2/36, \quad p_Z(4) = 3/36, \quad p_Z(5) = 4/36, \quad p_Z(6) = 5/36, $$ and so on (plotting this may be instructive; see the BH textbook, for example.) We can see from this example that we need to be careful when computing distributions of random variables like $X+Y$ from the distributions of $X$ and $Y$. It's certainly not that true that the distribution of $X+Y$ is $p_X+p_Y$!
It's worth noting, explicitly, that saying $X$ and $Y$ are identically distributed is not the same as saying $X=Y$. As we see in the example above, $X$ and $Y$ are clearly identically distributed, but saying $X=Y$ means that these two variables always have the same value which is not the case — our two dice rolls could have different outcomes!
The notion of independence for random variables is an intuitive adaptation of the definition for events. We want to say that random variables $X,Y$ are independent if knowing the outcome of one never affects the distribution of the other. The next definition makes this precise.
We say that random variables $X,Y$ are independent if for all $x,y\in\mathbb{R}$, $$ \Pr(X=x, Y=y) = \Pr(X=x)\Pr(Y=y). $$ Mutual independence is defined analogously: $X_1,\ldots,X_n$ are mutually independent if for all $x_1,\ldots,x_n$, the events $X_i=x_i$ are mutually independent.
The notation $\Pr(X=x, Y=y)$ means the probability that $X=x$ and $Y=y$. It is less clunky than trying to use a "$\cap$" or other notation. The definition for more than two random variables is a direct adaptation of mutual independence for several events.
If $X$ and $Y$ represent distinct independent fair die rolls, then (by fiat) they are independent — one die roll's outcome cannot affect the other's! So for any $x,y\in\{1,2,3,4,5,6\}$ we have $$ \Pr(X=x, Y=y) = \frac{1}{36} = \frac{1}{6}\cdot\frac{1}{6}= \Pr(X=x)\Pr(Y=y). $$ If we let $S=X+Y$ and $D=X-Y$, we might have an intuition that these variables are not independent. We can show this by finding some values $s,d$ for which $\Pr(S=s, D=d) \neq \Pr(S=s)\Pr(D=d).$ For instance, $\Pr(S=7, D=0) = 0$ since no die roll can have difference zero and sum to an odd number, but $\Pr(S=7)\Pr(D=0) \gt 0$ because these events individually have non-zero probability. Note that it was sufficient to find one combination of $s,d$ for which these random variables violated Definition 8.9.
When studying probability, the same distributions tend to show up over and over again. Thus several distributions are given names and a compact notation is used to indicate that a random variable has a commonly-known distribution.
A random variable $X$ is said to have the uniform distribution over $S$, written $X\sim\mathrm{U}(S)$, if $\Pr(X=s)=\frac{1}{|S|}$ for all $s \in S$.
This should be a very familiar distribution - it's the one have been using and relying on in all of our previous work with probability. After all, we've been assuming that all relevant outcomes in our sample space have equal probability of occurring - this just formalizes that.
A random variable $X$ is said to have the Bernoulli distribution with parameter $p$, written $X\sim\mathrm{Bern}(p)$, if $\Pr(X=1)=p$ and $\Pr(X=0) = 1-p$.
Bernoulli random variables are frequently described as trials that either succeed (i.e. output $1$) or fail (i.e. output $0$).
Suppose we roll a fair six-sided die, and define $$ X(\omega) = \begin{cases} 1 & \omega \leq 3 \\ 0 & \omega > 3 \end{cases}. $$ Then $X\sim \mathrm{Bern}(1/2)$. If we define $$ Y(\omega) = \begin{cases} 1 & \omega \leq 2 \\ 0 & \omega > 2 \end{cases} $$ then $Y\sim \mathrm{Bern}(1/3)$.
Bernoulli random variables will show up a few times later. For now we turn to another common distribution that models repeated independent Bernoulli trials. A simple physical example of this is tossing a coin $n$ times and counting the number of Heads that are shown. Each toss can be seen as a trial, and the trials are independent (which we declare by fiat, not calculation). This case is modeled by the binomial distribution, defined next.
A random variable $X$ is said to have the binomial distribution with parameters $n$ and $p$, written $X\sim\mathrm{Bin}(n,p)$, if for $k=0,1,\ldots,n$, $$ \Pr(X=k)=\binom{n}{k}p^k(1-p)^{n-k}, $$ and $\Pr(X=k)=0$ for all other $k\in\mathbb{R}$.
This is just a definition, but it deserves some explanation. The parameter $n$ corresponds to the number of trials, and $p$ corresponds to the probability of success of each trial. To see why the probabilities are assigned this way, consider first some sequence of successes and failures that describes the aggregate outcome of the $n$ trials; For instance, with $n=5$ we may have the outcome $SFFSS$ to indicate that the second and third trials failed and the rest succeeded. This particular outcome should have probability $p(1-p)(1-p)pp=p^3(1-p)^2$, since the trials are independent. Indeed, this is probability of any particular outcome that has three successes (and thus two failures).
Now to calculate $\Pr(X=3)$, we need to count the number of outcomes with example three successes. There are $\binom{5}{3}$ such outcomes, which correspond to strings with three $S$'s and two $F$'s. Adding up these (disjoint) events, we justify that $\Pr(X=3)=\binom{5}{3}p^3(1-p)^2$. The general pattern is justified the same way.
Suppose we have a coin that is biased to show Heads with probability $3/4$. If we toss this coin $100$ times and let $X$ count the number of Heads, then $X\sim\mathrm{Bin}(100,3/4)$.
The following definition captures a notion of "weighted average" for random variables. It is extremely useful in understanding random variables, especially complicated ones where just knowing a formula for their distribution is not directly enlightening.
The expectation (or expected value or mean) of a random variable $X$, denoted $E(X)$, is $$ E(X) = \sum_{x\in\mathbb{R}} x \Pr(X=x). $$
If $X$ is a random variable and $g: \mathbb{R} \rightarrow \mathbb{R}$ is a function, then $$ E(g(X)) = \sum_{x\in\mathbb{R}} g(x) \Pr(X=x). $$
If $X$ is a fair die roll, then $$ E(X) = 1\cdot \frac{1}{6} + 2\cdot \frac{1}{6} +3\cdot \frac{1}{6} +4\cdot \frac{1}{6} +5\cdot \frac{1}{6} + 6\cdot \frac{1}{6} = 3.5. $$
If $X\sim\mathrm{U}(S)$ where $S = \{s_1, \cdots, s_n \}$, then $E(X) = \frac{s_1 + \cdots + s_n}{n}$.
$X$ takes the $n$ values $s_1, \cdots, s_n$ with equal probability of $\frac{1}{n}$, so $$ E(X) = s_1 \cdot \frac{1}{n} + \cdots + s_n \cdot \frac{1}{n} = \frac{s_1 + \cdots + s_n}{n}.$$ This proves the theorem.
If $X$ is the number of Heads obtained (0 or 1) from flipping a coin biased to Heads $\frac{3}{4}$ of the time, $$ E(X) = 0 \cdot \Pr(X=0) + 1 \cdot \Pr(X=1) = 0 \cdot \frac{1}{4} + 1 \cdot \frac{3}{4} = \frac{3}{4}. $$
If $X\sim\mathrm{Bern}(p)$, then $E(X) = p$.
The only values that $X$ takes with non-zero probability are $0$ and $1$, so $$ E(X) = 0\cdot\Pr(X=0) + 1\cdot\Pr(X=1) = \Pr(X=1) =p.$$ This proves the theorem.
The core trick deals with expectations of random variables that are sums of other random variables. Consider computing the expectation of the sum of two fair dice; This could be written $E(X+Y)$, where $X,Y$ are the die rolls. Using the formula for expectation, we get $$ E(X+Y) = \sum_{k=2}^{12} k\cdot \Pr(X+Y=k) = 7. $$
This is not so bad, but it is a small case of a more general issue: In order to compute this expectation, we need the distribution of $X+Y$ (i.e., we need to calculate $\Pr(X+Y=k)$), which is often more complicated that the distributions of $X$ and $Y$.
The following theorem gives us a way to compute $E(X+Y)$ without computing the distribution of $X+Y$. The property it establishes is known as linearity of expectation.
For any random variables $X,Y$ $$ E(X+Y) = E(X)+E(Y), $$ and if $c\in\mathbb{R}$, then $$ E(cX) = c\cdot E(X). $$
Our main goal is understand this intuitively and see how to apply it. The proof, which we do at the end of these notes, will be optional. (That proof states and uses Proposition 8.36, also proved at the end of the notes)
The BH textbook provides another intuitive explanation for how $E(X+Y)$ and $E(X)+E(Y)$ correspond to two different ways of computing averages of the sum of two lists. Here is a small example:
| $\omega$ | $X(\omega)$ | $Y(\omega)$ | $X(\omega) + Y(\omega)$ |
|---|---|---|---|
| $\omega_1$ | $2$ | $5$ | $7$ |
| $\omega_2$ | $10$ | $20$ | $30$ |
| $\omega_3$ | $0$ | $-1$ | $-1$ |
If we average $X$ and $Y$ separately (i.e. sum them and divide by $3$), their averages are $4$ and $8$, which sum to $12$. But if we add the lists and then take an average, we also get $12$, agreeing with the linearity of expectation.
The formula for linearity of expectation extends to more random variables by an easy induction. That is, for any random variables $X_1,\ldots,X_n$, $$ E(X_1+\cdots+X_n) = E(X_1)+\cdots+E(X_n). $$
Returning to the case of rolling two dice, We have that $E(X+Y)=7$ because $E(X)=E(Y)=3.5$, as previously calculated. In fact, if we roll $n$ dice, we know the expectation is $3.5n$, for any $n$, despite not computing the complicated distribution of the experiment.
Now we'll give the nice way to calculate the expectation of a binomial random variable.
If $X\sim\mathrm{Bin}(n,p)$, then $E(X)=np$.
If $X\sim\mathrm{Bin}(n,p)$, then $X$ can be modeled as the number of successful trials (from $n$ total trials) where each trial is independent and succeeds with probability $p$. Let $X_i$ be the random variable representing the success of trial $i$. Then $X = X_1 + \cdots + X_n$, so by linearity of expectation, we have \begin{align*} E(X) & = E(X_1 + \cdots + X_n) \\ & = E(X_1) + \cdots + E(X_n) \\ & = p + \cdots + p \\ & = np. \end{align*}
The real magic of linearity is that you can often apply it even when the original random variable is not explicitly the sum of other random variables. The trick is that you can decompose the random variable yourself, and then compute the simpler expectations and add them.
A common version of this trick is called the indicator method, where one writes a random variable as a sum of zero/one random variables, called indicators.
Let $A\subseteq \Omega$ be an event. Then the random variable $I_A$ defined by $$ I_A(\omega) = \begin{cases} 1 & \omega\in A \\ 0 & \omega\notin A \end{cases} $$ is called the indicator random variable for the event $A$.
Indicators are simply Bernoulli random variables with parameter $p=\Pr(A)$. Thus the expectation is easy to calculate:
If $I_A$ is the indicator random variable for $A$, then $E(A)=\Pr(A)$.
The Indicator Method works as follows. To compute $E(X)$, one proceeds:
Suppose we toss a coin 100 times, and let $X$ be the number of times we get five consecutive tosses $HTHTH$. The random variable $X$ is very complicated, and we won't even try to compute its distribution, but observe that the streaks counted by $X$ may overlap!
To compute $E(X)$, we create indicators $I_j$ that indicate if the streak $HTHTH$ appears starting at the $j$-th toss. Then $$ X = I_1 + \cdots + I_{96}. $$ Note that we stop at $96$ because the streak can't possibly start in the last four tosses. We have $E(I_j)=1/2^5$ since this simply the probability that we get a particular outcome in $5$ tosses. Thus the expectation is $E(X)=96/2^5=3$.
While expectation is a useful piece of information about a random variable, it does not tell the full story. For instance, suppose $X$ is a uniformly distributed random variable between the values of $1$ and $-1$, and $Y$ is another uniformly distributed random variable between the values of $100$ and $-100$. Then $E(X)=E(Y)=0$, but the behavior of these random variables is clearly different: $Y$ has bigger "swings", and for many applications the size of the "swings" is as important as the average value. The measure of "swingy-ness" is called variance and is defined in the first half of this lecture.
The following definition is the most common measure of swingy-ness of a random variable.
Let $X$ be a random variable with $E(X)=\mu$. The variance of $X$ is defined to be $$V(X) = E((X - \mu)^2).$$ The standard deviation of $X$ is $\sigma(X) = \sqrt{V(X)}$. Because of this, variance is sometimes denoted by $\sigma^2(X)$.
Let's break this definition down. First, the square is on the inside of expectation: It's not $(E(X-\mu))^2$. Indeed, that value is not interesting, because by linearity we have $$ (E(X-\mu))^2 = (E(X)-\mu)^2 = (\mu - \mu)^2 = 0. $$ (There, we used $E(\mu)=\mu$, which is that the expectation of a constant is just the constant.)
The square is in the definition so that the positive and negative swings do not cancel each other out. By squaring, all of the deviations become positive, and thus accumulate in the total measure of swings. A very natural question is: Why squaring? Why not raising to another larger even power? Or more intuitively, why don't we use the value $$ E(|X-\mu|) $$ which also has the effect of accumulating both down and up swings into the total? The best answer at this point is that the definition with the square is way easier to compute, and still close enough to being intuitively useful. A deeper answer is that this definition has many other theoretical and practical uses beyond getting intuition for swings.
The definition of variance is rarely used directly in computations. The following equivalent version is easier to use.
If $X$ is a random variable then $V(X) = E(X^2) - E(X)^2$.
Let $E(X)=\mu$. By linearity, \begin{align*} V(X) &= E((X-\mu)^2) \\ &= E(X^2 - 2\mu X + \mu^2) \\ &= E(X^2) - 2\mu E(X) + E(X)^2 \\ &= E(X^2) - 2\mu^2 + \mu^2 \\ &= E(X^2) - \mu^2. \end{align*}
This formula reduces the job of computing $V(X)$ to computing $E(X^2)$ and $E(X)$.
Recall the roll of a fair 6-sided die and that $E(X) = 3.5$. To compute the variance, we need $E(X^2)$. For $i = 1, 2, \dots, 6$, $X^2 = i^2$ with probability $\frac 1 6$. Then $$E(X^2) = \sum_{i=1}^6 \frac{i^2}{6} = \frac{91}6.$$ Then we can calculate $V(X)$ by $$ V(X) = E(X^2) - E(X)^2 = \frac{91}{6} - \left( \frac 7 2 \right)^2 = \frac{35}{12} \approx 2.92. $$ From this, we can also get the standard deviation, $\sqrt{V(X)} \approx 1.71$.