Last lecture, we introduced combinatorics, the study of counting the sizes of sets. We came up with the following approaches to count the number of ways to make $k$ choices from $n$ options:
| Order of choices matters | Order of choices doesn't matter | |
|---|---|---|
| Sampling options without replacement | $\frac{n!}{(n-k)!}$ | $\binom{n}{k}$ |
| Sampling options with replacement | $n^k$ | ???? |
| Balls are distinguishable | Balls are not distinguishable | |
|---|---|---|
| Up to one ball per bin | $\frac{n!}{(n-k)!}$ | $\binom{n}{k}$ |
| Any number of balls per bin | $n^k$ | ???? |
What about the last square? We'll see now that the last square will require a more sophisticated approach.
It'll help for us to ground our discussion in an example where we are making $k$ choices from $n$ options, where the order of our $k$ choices doesn't matter and we are sampling our $n$ options with replacement.
Suppose you're ordering six scoops of ice cream and there is a choice of four types, say cookies & cream, pralines & cream, salted caramel, and gold medal ribbon. This ice cream shop is well-stocked, so we can order as many scoops of each flavor as we'd like, but we don't care about the order of the flavors on our cone.
If our decision process goes one scoop at a time, we end up overcounting by an amount that depends on the decision we've made, so it's not easy to correct for it. If our decision process goes one flavor at a time instead, we end up with an unclear number of choices for each step after the first one. Either way, the strategies we have developed so far won't quite work for us.
Let's try something different. One way to think about the number of possible choices to make is to think about the number of ways to represent a distinct choice. Let's consider one possible selection, $C,P,G,C,C,G$ (three cookie, one praline, two gold), assuming this is the order in which we chose our scoops. However, since this is a combination and some of the elements are indistinguishable anyway, the order doesn't really matter, so let's group them together into $CCCPGG$. To be extra clear, let's indicate where one flavor ends and another flavor starts: $CCC|P|GG$.
Something that is implicitly represented in this notation is that we chose 0 scoops of salted caramel. Following the order of our flavor list (C, P, S, G), I can add another line to indicate 0 scoops of S: $CCC|P||GG$. Now that I've established that I will always write the scoops in this order, I don't actually need their letter labels, so I can equivalently write $***|*||**$.
Let's consider another possible choice: $*||*|****$, which is one cookies & cream, one salted caramel, and four gold medal ribbon. What we observe is that each choice of six items from four classes can be represented by an arrangement of six stars representing the items and three bars representing the division of classes of items.
But this is something we've already seen before: it's just a string problem over the alphabet $\{*,|\}$. Since we have six objects and four classes, we can view our possible selections as a string of length 9 with 6 $*$s and 3 $|$s and ask how many such strings there are. There are $$\binom 9 6 = \binom 9 3 = \frac{9!}{6!3!} = \frac{9 \cdot 8 \cdot 7}{3 \cdot 2 \cdot 1} = 3 \cdot 4 \cdot 7 = 84$$ such strings.
This method of using stars and bars to denote the objects and categories was popularized by William Feller's An Introduction to Probability Theory and its Applications in 1950.
There are $\binom{n+k-1}{k} = \binom{k+n-1}{n-1} = \frac{(n+k-1)!}{k!(n-1)!}$ ways to make $k$ selections from $n$ options, where order of selections does not matter and a option may be selected multiple times.
We can represent each possible outcome as a string of length $k+n-1$ over the alphabet $\{\star,|\}$, with $k$ stars representing the $k$ selections to be made and $n-1$ bars demarcating which of the $n$ options was selected. Then there are $\binom{k+n-1}{k}$ possible ways to choose spots for the $k$ $\star$s. Since all remaining spots must be occupied by $|$s, this is the same as choosing spots for $n-1$ $|$s, and there are $\binom{n+k-1}{n-1}$ ways to do so.
We can now fill out our completed table, representing the number of ways to throw $k$ balls (representing $k$ choices to be made) into $n$ bins (representing the $n$ options overall), we have:
| Balls are distinguishable | Balls are indistinguishable | |
|---|---|---|
| At most one ball per bin | $\frac{n!}{(n-k)!}$ | $\binom{n}{k}$ |
| Any # of balls per bin | $n^k$ | $\binom{n+k-1}{k}$ |
The beauty and the difficulty of combinatorics is that almost every problem is about twisting the situation (through changed perspectives, the tricks we discussed earlier, or just creative analogies) into one that consists of a tractable decision process and/or a situation in our table above. It takes a ton of practice to learn when different analogies and different strategies are reasonable, and it is completely natural to try a bunch of wrong ones before stumbling upon one that works. Nobody ever said combinatorics was easy, but getting good at this will translate really well to becoming an adept problem-solver in any context!
Knowing all possible outcomes in a situation allows us to compute the probability of specific outcomes, which we'll focus on for most of the rest of this unit. Probability has wide applications throughout the field of computer science: cryptography is probabilistic, machine learning is probabilistic, and even modern data structures rely heavily on probabilistic guarantees to make them efficient.
Probability models all sorts of phenomena related to unpredictability and uncertainty. We are going to start with a simple class of random experiments: Those where every outcome is equally likely. We'll call this Naive Probability and later come back to think about situations where outcomes aren't equally likely.
To model a random experiment, one first defines a set of possible outcomes.
A sample space for a random experiment is a set $\Omega$, the members of which correspond to the possible outcomes.
Once we have a sample space, we could stop there and just consider individual outcomes. However, much of the richness of probability comes from using sets of outcomes, which correspond to English-language descriptions of features of outcomes. This motivates the following definition.
Let $\Omega$ be the sample space for a random experiment. An event is a subset of $\Omega$.
These examples use $\Omega_1,\ldots,\Omega_4$ from the previous example.
Events can be combined via unions, intersections, and complements to give new events. These operations have a very intuitive interpretation: If $E_1,E_2 \subseteq \Omega$ are events on the same sample space, then $E_1 \cup E_2$ represents "$E_1$ or $E_2$", $E_1 \cap E_2$ represents "$E_1$ and $E_2$", and $E_1^c$ represents "not $E_1$". (The complement is taken with respect to the universe $\Omega$.)
(This example is from [BH].) Suppose we flip $10$ coins. The sample space will be all length-$10$ strings consisting of $H$ and $T$ letters. That is, $\Omega$ contains strings like $HTTHTTTHHT$.
Finally, we define the probability of an event.
Let $\Omega$ be the sample space for a random experiment and $E\subseteq \Omega$ be an event. The (naive) probability of $E$ is defined to be $$ \mathrm{Pr}(E) = \frac{|E|}{|\Omega|}. $$
Of course, for this to make sense, $\Omega$ must be finite. You can work through some of the small examples above to calculate probabilities by hand, but to do the larger ones you'll need to infer the sizes of the events and samples spaces indirectly. This explains why counting is the main technical challenge in naive probability: everything depends on your ability to determine the sizes of sets!
Let's start by thinking about a problem that won't go the way we expect if we just naively approach it with the addition and multiplication rules from last lecture. Consider the following problem: I flip three coins. What is the probability that I get at least one heads?
One approach to this problem is the following. Denote by $H$ the event that I get at least one heads, and consider the following events:
Then, by definition, $H = X \cup Y \cup Z$. After all, $H$ can only occur if one of $X$, $Y$, or $Z$ occur! The addition rule then tells us that $$P(X \cup Y \cup Z) = P(X) \cup P(Y) \cup P(Z) = \frac{1}{2} + \frac{1}{2} +\frac{1}{2} = \frac{3}{2}. $$ So the probability then is 150%?!
This is obviously bad: nothing can have more than 100% chance to occur! So our really naive approach doesn't quite work out as well as we'd hope - let's look at some tricks we have to sharpen our calculation.
The reason that the prior computation is probably pretty clear to you as well - the three events as defined are overlapping! If we write out the outcomes counted by each event, we have
Looking at it this way, the union of the sets is clearly not the direct sum of their individual sizes - after all, all three sets have overlapping terms with each other.
To figure out how to address this, let's take a look at the following venn diagram between two sets:
One thing we could do here is to simply count each section separately: $$|X \cup Y| = |X \cap \overline Y| + |X \cap Y| + |\overline X \cap Y|.$$ This approach works, but it ends up getting pretty unwieldy with all these intersections, especially if $X$ and $Y$ are complex events.
A better approach would simply be to determine the size of the overlap and account for it. Note that if we had approached it naively we would have added $|X| + |Y|$, but then the central region ($|X \cap Y|$) is counted twice. In that case, to account for this mistake, we can simply subtract that region's size once to make sure we're no longer overcounting, telling us that $$|X \cup Y| = |X| + |Y| - |X \cap Y|.$$
Let's think about the three-set case now.
A similar approach to last time will get us almost there: if we try $$|X| + |Y| + |Z|,$$ we find that the outer regions are counted once, the regions in two circles are counted twice, and the center region is counted three times. Subtracting the intersections as follows $$|X| + |Y| + |Z|- |X \cap Y| - |X \cap Z| - |Y \cap Z|,$$ we have removed the overcounting problem, but note that we have now subtracted the central region three times as well! In terms of our example problem, the case HHH is in every single one of $|X|, |Y|, |Z|, |X \cap Y| , |X \cap Z|,$ and $|Y \cap Z|,$ so our current expression actually adds it three times and the subtracts it three times, which means we're not counting it in the end. To address this problem, we have to add it back: $$|X \cup Y \cup Z| = |X| + |Y| + |Z|- |X \cap Y| - |X \cap Z| - |Y \cap Z| + |X \cap Y \cap Z|.$$
You may notice a pattern here: we're subtracting even-termed intersections and adding odd-termed intersections. This is no coincidence.
The size of the union of sets can be computed as follows: $$\left|\bigcup_{i=1}^n A_i\right| = \sum_{j=1}^n (-1)^{j+1}\left(\sum_{S \subseteq \{1...n\}, \: |S| = j} \left|\bigcap_{s \in S}A_s\right|\right).$$ For two sets, this evaluates to $$|X \cup Y| = |X| + |Y| - |X \cap Y|.$$ For three, it evaluates to $$|X \cup Y \cup Z| = |X| + |Y| + |Z|- |X \cap Y| - |X \cap Z| - |Y \cap Z| + |X \cap Y \cap Z|.$$
This is provable by induction on $n$ if you are interested, though for the purposes of this class you only really need to know that it follows this pattern.
It's also worth noting that if the sets are disjoint, this summation actually resolves back down to the original addition rule!
To wrap up our problem: there are four possibilities for $|X|$ (or any of the other individual set), two possibilities for $|X \cap Y|$ (because the last coin can be H or T), and one possibility for $|X \cap Y \cap Z|$ (only the outcome HHH), leaving us with, by PIE, $$|X \cup Y \cup Z| = 4 + 4 + 4 - 2 - 2 - 2 + 1 = 7.$$ There are $8$ possibilities in total for three coinflips, giving us a probability of $7/8$.
Here's a different trick we could've employed to make solving this problem even faster. Something you may have noticed in the prior strategy is that the entire sample space consists of four possibilities - the three we mentioned, plus the event where I flip zero heads. Specifically, if we define $D$ to be the event where we flip zero heads, then the total sample space is $\Omega = A \cup B \cup C \cup D,$ or substituting we find $\Omega = H \cup D$. Because these events are disjoint, we also get that $|\Omega| = |H| + |D|$, or more helpfully for this problem $|H| = |\Omega| - |D|$. Instead of dealing with all the possibilities where we flip at least one heads, we can save ourselves a lot of trouble by considering the complement, which in this case is flipping no heads!
The benefit is that, for this problem in particular, the complement is super easy to compute. After all, for any $n$ coins, there is only one outcomes that results in flipping zero heads. For our three coins, there are $8$ possible outcomes, leaving us with $|H| = |\Omega| - |D| = 8 - 1 = 7$, in a much sleeker approach than either of the previous ones.
This trick of counting the complement is a strategy that is not too commonly useful but can be incredibly powerful when it works.
If $U, A$ are finite sets, and $A \subseteq U$, then $$|U \setminus A| = |U| - |A|.$$ In the language of complements, this is $$|A^c| = |U|-|A|$$ or, equivalently, $$|A| = |U|-|A^c|,$$ where $U$ is the universe under consideration.
I also consider this self-evident, but one can give a proof from the Addition Rule.
Let us count the number of $6$ character passwords consisting of lowercase letters and digits, that have at least one digit.
First, let's try to use a standard decision process. If we try that approach, then the first option has $36$ choices. But how many options does the second choice have? Well, it's still $36.$ But if you can continue until the end sixth choice, when your number of choices will depend on the past: If you picked a digit in one or more of the first five spots, then you have $36$ options. If not, then you must pick a digit, and you only have $10$ options.
The result is a tree that is not regularly shaped: At the last level, some nodes will have $36$ branches and some will have only $10$. This means we can't apply the Multiplication Rule, at least not directly.
Instead, let's do this by counting the compliment: How many passwords are there that do not contain a digit? That's easy, those are just passwords with all letters, and there are $26^6$ of them. The set of all passwords consisting of letters and digits has size $36^6$, as there are $36$ options for each of the $6$ characters. Putting this together gives $36^6-26^6$.
There are $10$ people are auditioning for a play there are $7$ total roles. How many ways are there to assign people to roles, assuming any person can play any role?
Let's first map this to the balls and bins analogy. Here, we can consider the roles to be balls, and people to be bins. We're trying to determine the number of ways to assign balls to bins, where balls are distinguishable and at most one ball can fit in each bin.
Our decision process is simple: for each ball (role), we select a previously unselected bin (person). Then the number of possibilities for the first decision is $10$, then $9$, and so on until $4$ for the last role. In general, the number of possibilities is $$n \cdot (n-1) \cdot (n-2) \cdot \cdots \cdot (n-k+1) = \frac{n!}{(n-k)!}$$
There are $8$ people at a restaurant. Each one of them is selecting a dinner entree from a menu with $13$ selections. How many possibilities are there for what everyone orders?
Again, let's first map this to the balls and bins analogy. Here, we can consider the bins to be the entrees and the balls to be the people. Each ball (person) is different, and each bin (entree) can be taken by more than one ball (person), so we know we are in the bottom left square of the table above.
A decision process will help us here. Each ball (person) has $13$ bins (entrees) possible, so there are $13^8$ possibilities in total. More generally, there are $n^k$ possible outcomes.
For your major, you have to take $3$ elective courses. There are a total of $20$ possible elective courses to choose from. How many different combinations of three electives can you take?
In this case, we have $3$ balls (courses taken) that we are distributing among $20$ bins (possible courses). You can't take a course more than once (at least, not in this scenario), so we are in the top right section of our table.
To actually count the possibilities, we note that this is a straightforward case of us choosing subsets of size $3$ from a set of size $20$. Thus, the number of options is $\binom{20}{3}$, or more generally $\binom{n}{k}$.