$union
[1] 2 4 6 5
$intersection
[1] 6
$complement
[1] 1 3 518 Probability
Every model is a claim about uncertainty. A prediction that does not say how confident it is has not said very much, and a model that cannot be wrong cannot be checked.
Probability is the arithmetic of uncertainty. This chapter builds it from three axioms to Bayes’ theorem, and then spends real time on the ways the arithmetic reliably defeats intuition — because in applied work that is where the mistakes are.
18.1 Reasoning under uncertainty
There are two standard readings of the statement “this coin has probability 0.5 of landing heads”.
Frequentist: in a long run of flips, the proportion of heads tends to 0.5. A probability is a limiting frequency, and only repeatable events have one.
Bayesian: 0.5 is a degree of belief, updated as evidence arrives. On this reading it is meaningful to say “there is a 70% chance this patient has the disease”, even though the patient will not be resampled.
The mathematics is identical either way — the axioms below do not care — and the split is about what probabilities mean, not how they combine. Most applied work moves between them without comment, and this book will too.
18.2 Sample spaces and events
The sample space \(\Omega\) is the set of all possible outcomes. An event is a subset of it — so everything in Chapter 3 applies directly.
| Experiment | \(\Omega\) |
|---|---|
| one coin flip | \(\{H, T\}\) |
| one die roll | \(\{1,2,3,4,5,6\}\) |
| two coin flips | \(\{HH, HT, TH, TT\}\) |
| a person’s height | \((0, \infty)\) |
For a die roll, “even” is the event \(A = \{2,4,6\}\) and “at least 5” is \(B = \{5,6\}\). Set operations become the language of combining events:
| Set notation | Reads as |
|---|---|
| \(A \cup B\) | \(A\) or \(B\) |
| \(A \cap B\) | \(A\) and \(B\) |
| \(A^c\) | not \(A\) |
| \(A \cap B = \emptyset\) | \(A\) and \(B\) are mutually exclusive |
18.3 The axioms of probability
Everything follows from three rules, due to Kolmogorov:
\[ \begin{aligned} &P(A) \geq 0 &&\text{probabilities are non-negative} \\ &P(\Omega) = 1 &&\text{something happens} \\ &P(A \cup B) = P(A) + P(B) \quad\text{if } A \cap B = \emptyset &&\text{additivity} \end{aligned} \tag{18.1}\]
That is all. Every other rule is a consequence:
\[ \begin{aligned} P(A^c) &= 1 - P(A) \\ P(\emptyset) &= 0 \\ P(A \cup B) &= P(A) + P(B) - P(A \cap B) \\ A \subseteq B &\;\Rightarrow\; P(A) \leq P(B) \end{aligned} \]
The third deserves attention: for events that are not mutually exclusive you must subtract the overlap, or it is counted twice. Forgetting that term is one of the most common errors in elementary probability.
P_A P_B P_A_and_B P_A_or_B
0.5000000 0.3333333 0.1666667 0.6666667 \(\{2,4,5,6\}\) has four outcomes out of six, which is \(2/3\) — and the inclusion–exclusion formula gets it right where naive addition would have given \(5/6\).
18.4 Counting
When outcomes are equally likely, probability reduces to counting:
\[ P(A) = \frac{|A|}{|\Omega|} \]
Two counting rules cover most cases. Choosing \(k\) items from \(n\) with order mattering gives permutations, \(n!/(n-k)!\); without order gives combinations,
\[ \binom{n}{k} = \frac{n!}{k!\,(n-k)!} \]
poker_hands two_heads_in_three
2598960.000 0.375
NoteIn machine learning
Counting arguments set the size of a hypothesis space, which is the crude version of model capacity. With \(p\) binary features there are \(2^p\) possible inputs and \(2^{2^p}\) possible classifiers on them — so for \(p = 10\) there are more candidate classifiers than atoms in the observable universe. Generalization is possible only because we search a tiny structured corner of that space.
18.5 Conditional probability
Knowing that \(B\) happened changes what you should believe about \(A\):
\[ P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0 \tag{18.2}\]
Read \(P(A \mid B)\) as “the probability of \(A\) given \(B\)”. The formula says: restrict attention to the world where \(B\) happened, then ask what fraction of it also has \(A\). The division by \(P(B)\) is renormalization — that smaller world has to add up to 1 again.
Worked example. Roll a die. \(A\) = “even”, \(B\) = “at least 5”.
\[ P(A \mid B) = \frac{P(\{6\})}{P(\{5,6\})} = \frac{1/6}{2/6} = \frac{1}{2} \]
p_AB / p_B[1] 0.5Rearranging Equation 18.2 gives the multiplication rule, which is often the more useful form:
\[ P(A \cap B) = P(A \mid B)\,P(B) = P(B \mid A)\,P(A) \tag{18.3}\]
WarningWatch out
\(P(A \mid B)\) and \(P(B \mid A)\) are different quantities and confusing them is the single most consequential error in applied probability.
The probability that a test is positive given disease is not the probability of disease given a positive test. In court, the probability of the evidence given innocence is not the probability of innocence given the evidence — a mistake with a name, the prosecutor’s fallacy, and real convictions attached to it.
18.6 Independence
Events are independent when knowing one tells you nothing about the other:
\[ P(A \cap B) = P(A)\,P(B) \tag{18.4}\]
Equivalently \(P(A \mid B) = P(A)\): conditioning changes nothing.
Worked example. Roll two dice. Let \(S_7\) be “the sum is 7” and \(F_1\) be “the first die shows 1”.
\(P(S_7) = 6/36 = 1/6\), and \(P(S_7 \mid F_1) = 1/6\) as well — whatever the first die shows, exactly one value of the second makes the sum 7. So they are independent, which surprises most people.
Now let \(S_8\) be “the sum is 8”. Then \(P(S_8) = 5/36\), but \(P(S_8 \mid F_1) = 0\): with a 1 showing, the second die would need to be 7. Not independent.
grid <- expand.grid(d1 = 1:6, d2 = 1:6)
p_of <- function(cond) mean(cond)
c(
P_sum7 = p_of(grid$d1 + grid$d2 == 7),
P_sum7_given_first1 = p_of(
(grid$d1 + grid$d2 == 7)[grid$d1 == 1]
),
P_sum8 = p_of(grid$d1 + grid$d2 == 8),
P_sum8_given_first1 = p_of(
(grid$d1 + grid$d2 == 8)[grid$d1 == 1]
)
) P_sum7 P_sum7_given_first1 P_sum8 P_sum8_given_first1
0.1666667 0.1666667 0.1388889 0.0000000 Independence is a property of the specific events, not of the underlying experiment. The dice are physically unrelated in both cases; whether two events built from them are independent depends on the events.
WarningWatch out
Mutually exclusive is not independent — in fact they are close to opposites. If \(A\) and \(B\) cannot both happen, then learning \(A\) occurred tells you \(B\) definitely did not, which is maximally informative. Mutually exclusive events with non-zero probability are always dependent.
18.7 The law of total probability
Split the sample space into mutually exclusive, exhaustive pieces \(B_1, \dots, B_k\) — a partition. Then any event’s probability is a weighted average over the pieces:
\[ P(A) = \sum_{i=1}^{k} P(A \mid B_i)\,P(B_i) \tag{18.5}\]
Each term is “the chance of \(A\) if we are in case \(i\)” times “the chance of being in case \(i\)”. This is the workhorse for problems where the answer depends on something unknown: condition on it, then average it out.
Worked example. A test is positive 99% of the time in sick people and 5% of the time in healthy people. If 1% of the population is sick, what fraction test positive?
\[ P(+) = (0.99)(0.01) + (0.05)(0.99) = 0.0099 + 0.0495 = 0.0594 \]
prev <- 0.01
sens <- 0.99 # P(+ | disease)
fpr <- 0.05 # P(+ | no disease)
p_pos <- sens * prev + fpr * (1 - prev)
c(
from_sick = sens * prev,
from_healthy = fpr * (1 - prev),
total = p_pos
) from_sick from_healthy total
0.0099 0.0495 0.0594 Notice already that most positives come from healthy people — 0.0495 against 0.0099 — purely because there are so many more of them. That observation is the whole of the next section.
18.8 Bayes’ theorem
Combining Equation 18.3 with Equation 18.5 gives the rule for reversing a conditional probability:
\[ P(B \mid A) = \frac{P(A \mid B)\,P(B)}{P(A)} \tag{18.6}\]
with the denominator usually expanded via Equation 18.5. The pieces have names:
| Term | Name | Meaning |
|---|---|---|
| \(P(B)\) | prior | belief before seeing the evidence |
| \(P(A \mid B)\) | likelihood | how well \(B\) explains the evidence |
| \(P(B \mid A)\) | posterior | belief after seeing the evidence |
| \(P(A)\) | evidence | normalizing constant |
Worked example. Same test. You test positive. What is the probability you are sick?
\[ P(D \mid +) = \frac{(0.99)(0.01)}{0.0594} = 0.1667 \]
posterior <- sens * prev / p_pos
posterior[1] 0.1666667About one in six. A test that is 99% sensitive and 95% specific leaves you more likely healthy than sick, because the prior was so low.
The clearest way to see it is in natural frequencies. Imagine 10,000 people:
n <- 10000
sick <- n * prev
true_pos <- sick * sens
false_pos <- (n - sick) * fpr
c(
sick = sick, healthy = n - sick,
true_positives = true_pos, false_positives = false_pos,
share_truly_sick = true_pos / (true_pos + false_pos)
) sick healthy true_positives false_positives
100.0000000 9900.0000000 99.0000000 495.0000000
share_truly_sick
0.1666667 99 against 495. The 5% false positive rate applies to 9,900 people, and \(0.05 \times 9900 = 495\) swamps the 99 true positives. The base rate is doing most of the work, and ignoring it is the base rate fallacy.
prevs <- seq(0.001, 0.3, length.out = 200)
post <- sens * prevs / (sens * prevs + fpr * (1 - prevs))
draw_line(
x = prevs,
y = post,
points = FALSE,
xlab = "prevalence",
ylab = "P(disease | positive)"
)
NoteIn machine learning
Bayes’ theorem is the engine of probabilistic modelling. A naive Bayes classifier applies Equation 18.6 directly with an independence assumption. Bayesian inference treats parameters as random and updates a prior into a posterior (Section 21.8). And the base rate lesson is why accuracy is a misleading metric on imbalanced data: a classifier that always says “healthy” is 99% accurate here and completely useless.
18.9 Common probability fallacies
Probability defeats intuition reliably enough that the failures have names.
Base rate neglect. Judging \(P(D \mid +)\) from the test’s accuracy alone, ignoring \(P(D)\). The section above is the canonical case.
The prosecutor’s fallacy. Treating \(P(\text{evidence} \mid \text{innocent})\) as \(P(\text{innocent} \mid \text{evidence})\). A one-in-a-million match in a city of ten million means about ten people match.
The gambler’s fallacy. Believing independent events “balance out”. After five heads, the next flip is still 50/50 — the coin has no memory. Its mirror image, the hot hand fallacy, reads streaks as evidence of a changed process.
The conjunction fallacy. Rating \(P(A \cap B)\) above \(P(A)\) because the conjunction tells a better story. It is never true: \(A \cap B \subseteq A\), so \(P(A \cap B) \leq P(A)\) always.
Confusing “rare” with “never”. Events of small probability are near-certain across many trials. The birthday problem is the standard demonstration.
[1] 0.1169 0.4114 0.5073 0.7063 0.9704 0.9992kk <- 1:70
draw_line(
x = kk,
y = sapply(kk, birthday),
points = FALSE,
xlab = "people in the group",
ylab = "P(some shared birthday)"
)Twenty-three people give \(\binom{23}{2} = 253\) pairs, and it is pairs that matter. The intuition fails because people instinctively compare against their own birthday, which is a different and much rarer question.
18.10 Summary
| Concept | Formula |
|---|---|
| Axioms | \(P \geq 0\), \(P(\Omega) = 1\), additivity |
| Complement | \(P(A^c) = 1 - P(A)\) |
| Union | \(P(A \cup B) = P(A) + P(B) - P(A\cap B)\) |
| Conditional | \(P(A\mid B) = P(A \cap B)/P(B)\) |
| Multiplication | \(P(A\cap B) = P(A\mid B)P(B)\) |
| Independence | \(P(A\cap B) = P(A)P(B)\) |
| Total probability | \(P(A) = \sum_i P(A\mid B_i)P(B_i)\) |
| Bayes | \(P(B\mid A) = P(A\mid B)P(B)/P(A)\) |
18.11 Exercises
1. A card is drawn from a standard deck. Find \(P(\text{heart})\), \(P(\text{face card})\), and \(P(\text{heart or face card})\).
TipSolution
13 hearts, 12 face cards, and 3 cards that are both.
p_h <- 13 / 52
p_f <- 12 / 52
p_hf <- 3 / 52
c(heart = p_h, face = p_f, either = p_h + p_f - p_hf) heart face either
0.2500000 0.2307692 0.4230769 \(22/52 \approx 0.423\). Adding without subtracting the overlap would give \(25/52\) and double-count the three face cards that are hearts.
2. Two dice are rolled. Given that the sum is 8, what is the probability the first die shows 5?
TipSolution
The sum-8 outcomes are \((2,6),(3,5),(4,4),(5,3),(6,2)\) — five of them, one with a 5 first.
outcomes_with_sum_8 P_first_is_5_given_sum_8
5.0 0.2 \(1/5\), not \(1/6\). Conditioning shrank the sample space to five equally likely outcomes, and the answer must be computed inside that smaller world.
3. A rarer disease: prevalence \(0.1\%\), same test. Now what is \(P(D \mid +)\)?
TipSolution
post_at <- function(p) sens * p / (sens * p + fpr * (1 - p))
c(
prevalence_1pct = post_at(0.01),
prevalence_0.1pct = post_at(0.001)
) prevalence_1pct prevalence_0.1pct
0.16666667 0.01943463 Under 2%. Making the disease ten times rarer makes a positive result roughly ten times less informative — the posterior is nearly proportional to the prior when the prior is small.
This is the quantitative case against mass screening for rare conditions: the test has not changed, but almost every positive it produces is now false.
4. Show that if \(A\) and \(B\) are independent, so are \(A\) and \(B^c\).
TipSolution
\[ \begin{aligned} P(A \cap B^c) &= P(A) - P(A \cap B) \\ &= P(A) - P(A)P(B) \\ &= P(A)\bigl(1 - P(B)\bigr) = P(A)P(B^c) \end{aligned} \]
P_ev_and_hi product P_ev_and_not_hi product_complement
0.1666667 0.1666667 0.3333333 0.3333333 Independence survives complementation. It also survives on the other side, and for both at once — which is what lets you build joint probabilities of independent events freely.
5. In the Monty Hall problem you pick one of three doors; the host, who knows where the car is, opens a different door revealing a goat and offers a switch. Should you?
TipSolution
Yes. Your first pick is right with probability \(1/3\), and that does not change — the host was always able to open a goat door, so doing so is not evidence about your door. The remaining \(2/3\) concentrates entirely on the one unopened door.
stay_wins switch_wins
0.33393 0.66607 Switching wins whenever the first pick was wrong, which is \(2/3\) of the time. The simulation needs no explicit host: “switch wins” is exactly “first pick was wrong”.
The intuition fails because the host’s action feels informative and is not — he had no choice but to reveal a goat.
6. How many people are needed for a \(95\%\) chance of a shared birthday?