07The Base-Rate Trap

Hand the flip a test that is 99% accurate and a disease that strikes 1 in 1000 people. The result comes back positive. The honest chance you are actually sick is under 10% — not 99%, not anywhere close. And nothing is wrong with the arithmetic. That is exactly the problem. We built the whole machine last chapter, and it now runs in both directions, cause to effect and effect back to cause. Right at the end we caught it doing this, and it should have made you uneasy. It is the same Bayes' theorem, with no new axiom and no new trick. We have simply walked it into its most extreme regime, where the prior is tiny and the answer stops matching anyone's gut. So this chapter is really one idea, felt from every angle. When the thing you are testing for is rare, the vast healthy majority manufactures more false positives than the tiny sick group makes true positives. A test everyone calls very accurate can turn out to be nearly useless. We'll make that crowd of false positives visible as a field of a thousand dots, so the shocking number becomes believable. We'll separate the two knobs everyone wants to mash into one, sensitivity and specificity, and show that neither one can be collapsed into a single "accuracy." And we'll meet the same trap wearing a lab coat in medicine and a hoodie in machine learning, where it's called the accuracy paradox. The maths is Chapter 6's. The stakes are the whole real world.

01The shock, made of people

Let's stand on the exact rung we left off on. Chapter 6 ended with Bayes' theorem in full: P(sick | positive) = P(positive | sick)·P(sick) / P(positive). All the drama hides in one innocent-looking piece, the denominator, which we called the evidence. It counts every way a positive can happen: some positives come from a sick person, and the rest come from a healthy person the test got wrong. Plug in a test that is 99% accurate and a disease that strikes 0.1% of people, and the honest posterior is about 9%. The arithmetic is airtight. The answer is absurd to the gut. So here is the naked truth about that right formula: almost nobody believes it. That disbelief is the gap we are about to cross.

Before I show you why that answer is believable, let's make the disbelief honest by committing to a guess. Forget the disease for a moment. Take the lecture's own example: a workplace drug test that is 90% accurate, used on a population where 10% of people really do use. You test one employee, and the result comes back positive. Don't compute anything yet. Just feel out a number. What is the chance this employee is actually a user? Fix that figure in your head, lock it in, and then reveal the truth.

The real answer is 50%, a coin flip. If you guessed somewhere near 90% you are in excellent company, and you are badly wrong: a positive on that drug test tells you essentially nothing. The reason is a headcount, not an equation, and this is the move that finally makes it click, so slow down here. Stop thinking in fractions and start counting people. Take a room of 1000 employees. Drag the three dials: how common the disease is, how well the test catches the sick, and how cleanly it clears the healthy. Watch the true positives and false positives light up as actual dots. The posterior is nothing more mysterious than this: of all the dots that lit up, what fraction belong to people who are truly sick?

There it is, and you can literally point at it: one true positive glowing inside a small crowd of false positives. The 9% stops being a paradox and becomes something you can count. It is just 1 out of 11 lit dots. That is the entire chapter in one picture. Everything that follows is us working out why the two crowd sizes come out the way they do, and what it means when they do. But first, notice what just happened to your intuition. The fraction version ambushed you. The count version disarmed the trap completely. That is not an accident, so watch the identical numbers rendered both ways at once.

Here's why the counts win. Reason in natural frequencies and every claim you make is a headcount: "1 in 1000, of whom this many, against 999 of whom that many" — the tiny sick group first, then the enormous healthy crowd. Said that way, the base rate is baked into the counts and comes along for free. Reason in bare conditional probabilities instead, and the base rate becomes a separate factor that you have to remember to include. A separate factor is easy to just... forget, and forgetting it is precisely the bug. The mistake even has a name, and it is an old friend from Chapter 6.

The whole error is one swapped arrow. You want the backward arrow, P(sick | positive): given a positive result, how likely is the disease? The mind quietly grabs the forward arrow instead, P(positive | sick), because that is the arrow the test advertises. In doing so, it drops the base rate on the floor. It answers the easy question and hopes the answer fits the hard one. Base-rate neglect is just the flip, left un-flipped. So the cure is to be ruthless about which direction you are in. And that starts with being ruthless about the two knobs a test actually has.

02Two knobs, never one

"It's 95% accurate" is a lie of omission, and people describe tests that way constantly. A test does not have one accuracy. It has two separate jobs on two separate populations, and it can be brilliant at one job while it botches the other. Let's name both correctly, once and for all, and settle a slip from earlier. Start with sensitivity, which is P(positive | sick): of the people who are sick, what fraction does the test catch? Specificity is P(negative | healthy): of the people who are healthy, what fraction does the test correctly clear? Build the population out on a grid and watch each definition light up its own column.

You just watched the grid light one column at a time, and watching is not the same as building. So here is a cold case with no lights to help you: 1000 people, of whom 80 are truly sick. The test flags 72 of those 80 sick people, and 46 of the 920 healthy ones. Drop each count into its own cell. The step that trips everyone is the very first one: which split is the truth about these people, and which is the test's call?

This grid is the confusion matrix, so read it carefully: every one of its four cells has a name you'll meet for the rest of your life. Down the sick column sit the true positive and the false negative. Down the healthy column sit the false positive and the true negative. Sensitivity lives entirely in the sick column, and specificity lives entirely in the healthy one. The two are computed on disjoint groups of people, which is exactly why one number can never stand in for both. That is not a quibble, so let me show it: here are two tests with the same overall accuracy and wildly different verdicts.

Look at what "accuracy" did there. It averaged the two populations together and hid which one the test fails on. Both tests post an identical 90% accuracy. One of them hands you a trustworthy posterior. The other hands you a coin flip. The only difference is how they split their errors across sensitivity and specificity. And when the disease is rare, one of those two knobs matters far, far more than the other. Hold sensitivity pinned high, drag specificity down by a few points, and watch the posterior fall off a cliff.

You just felt a small slip in specificity gut the posterior, while the same slip in sensitivity barely showed. Now stop watching and decide. You are the engineer. The disease is rare, only 1 in 100 people carry it, and you may add five points to exactly one knob. Which knob buys more trust in a positive result? Commit to an answer before you look. The distance between the two options is the lesson, turned into a choice.

Feel the asymmetry. Dropping specificity from 99% to 95% sounds tiny, and it guts the posterior: false alarms among the healthy jump from 1 in 100 to 5 in 100. The same wiggle in sensitivity barely registers. That is not a coincidence. It is structural, and once you see the mechanism you can never un-see it. Here is why.

03Why the base rate wins

Here is the engine of the entire trap, and each side of it fits on one line. The true positives are the sick people the test catches: (prevalence) × (sensitivity), a tiny group multiplied by a great hit-rate. The false positives are the healthy people the test wrongly flags: (1 − prevalence) × (1 − specificity), a huge group multiplied by a small error-rate. The whole drama is a fight between those two products. In 1000 people, a 0.1% disease and a 99% test give about 1 real case against 10 false alarms. Put them on the same scale and watch which one wins as you shrink the base rate.

Here is the crux, flat and plain: a small percentage of a huge number can dwarf a large percentage of a tiny number. Even a near-perfect test, firing on the vast healthy crowd, throws off enough false positives to bury the handful of real true positives that come from the small sick group. That is what the denominator from Chapter 6 was doing all along. The denominator is the evidence, the total-probability sum, and it fills up with the healthy crowd's contribution. The posterior is whatever is left. So the trap is not about the test being bad. It is about the base rate being small. Which means the trap must dissolve as the base rate grows. Slide the disease from rare to common and watch the paradox evaporate.

Watch the posterior climb as the disease gets commoner. At 0.1% prevalence, a positive means 9%. At 10% prevalence, the same test means 92%. By 50% prevalence it means 99%. Identical test, believable again, because the sick group is no longer outnumbered. So every test has a threshold where a positive result tips from probably wrong to probably right: the prevalence at which the posterior crosses 50%. Go back to the 90%-sensitive, 90%-specific drug test and predict where its crossing sits before you drag to it.

That crossover point is the honest summary of the whole regime. Below it, a positive result is more likely a false alarm than a real detection. No amount of staring at the test's advertised accuracy will tell you that. Only the flip, run with the real base rate, will. Now let's push both knobs to their breaking point and watch a "perfect" test reveal itself as a fraud.

04The always-positive trap

Here is the cheapest way to build a test that is 100% sensitive, one that has never once missed a sick person: make a test that always says positive. It cannot possibly miss a sick person, because it flags everyone. Its sensitivity is a perfect 100%. Its specificity is a catastrophic 0%, because it flags every healthy person alive too. So when someone brags that their test is 100% sensitive, ask them how. Watch what this one does to the posterior.

The posterior falls straight back to the base rate. The test told you nothing you didn't already know before you ran it. So you cannot buy safety with a single knob. A test that is all sensitivity and no specificity is just a machine for saying "yes," and a machine that always says yes carries no information at all. Now for the part that should make every ML engineer sit up, because it is the identical maths under a different name, with much bigger stakes. Consider a classifier trained to flag a rare event: fraud, a tumour, one bad transaction in a million.

This is the accuracy paradox, and it is the always-positive test mirror-imaged: the same fraud, run the other way. Say the event you are detecting happens 0.1% of the time. Now build a model that never predicts it, one that answers "healthy," "not fraud," "no" every single time. It is right about the other 99.9% of cases, so it scores a dazzling 99.9% accuracy while catching zero of the cases you actually care about. Its recall is 0%. It is worse than useless, and on a naive accuracy leaderboard it looks like a champion. That is why accuracy is the wrong number to optimize on imbalanced problems, and why the flip tells you the right ones.

Fig. 16 gave you two words, recall and precision, and let you swing them. But a beginner walks away holding two vocabulary items instead of two ideas, so look at what actually differs between them. Both scores start from the very same corner cell, the true positives. Recall divides that cell by itself plus the sick you missed; precision divides it by itself plus the false alarms you raised. One numerator, two neighbours. Pick the neighbour and you have picked the score.

File that pairing away. In medicine the two knobs are called sensitivity and specificity. In machine learning they are called recall and precision. The warning is identical in both: a single accuracy number, laid over a rare class, is a lie. Same trap, three fields, one flip. So what do you actually do when your test is honest but weak and the disease is rare? You don't throw the test away. You use it correctly.

05Screen, then confirm

Let's pin the exact number the way the lecture does, so nothing here is hand-waved. Take its drug-test figures: 90% sensitive, 90% specific, 10% users. Step through the three lines of the flip. In 100 employees, 9 users test positive and so do 9 non-users, so the answer is 50%. Now make the test worse on the healthy side, dropping specificity to 80%, and run it again. The false positives double to 18 against the same 9 real ones, so the posterior sinks to 1 in 3. Watch the plug-in happen one beat at a time.

A 33% posterior from a positive result is genuinely bad. Trust it and you would be wrong two times in three. But the lecture's own instinct is the right one: if you really needed to know, you wouldn't stop there. You would screen everyone with the cheap test, then confirm each positive with a second one. And here is the beautiful part: this is Chapter 6's belief-update loop, back for real work. The first test's posterior is not the end of the story. It becomes the prior for the second test. Run two independent tests in sequence and watch two mediocre results compound into near-certainty.

That ×99 you watched carry the odds forward is not a fluke of these particular numbers. It is the whole flip folded into one multiplier, and it has a name: the likelihood ratio. It is the sensitivity divided by the false-positive rate, so 99% caught among the sick against 1% wrongly flagged among the healthy gives 99% ÷ 1% = 99. Write your belief as odds rather than a probability and Bayes shrinks to one line: prior odds × LR = posterior odds. Better still, it shows why two tests compound. You simply multiply their two ratios. Drive it once by hand and the full fraction never has to come out again on a screening problem.

That is the resolution of the whole trap. A single weak test drowns in false positives. But a positive on two independent tests is a far rarer coincidence for a healthy person to produce by luck. The second flip starts from the elevated prior the first test left behind, and it vaults the posterior up toward certainty. Screen wide and cheap. Confirm narrow and hard. Now let's prove the posterior with no formula at all, which is the thing this course can do that no lecture can. Simulate a million people, tag each one sick or healthy by the base rate, run the noisy test on every one, and just count the true and false positives. Predict where the empirical posterior lands, then run it.

The empirical fraction of positives who are truly sick lands right on the number the flip predicted. There was no Bayes' theorem anywhere in that loop, just a million coin-tosses and a tally. That is the strongest kind of proof there is: the algebra and the brute-force headcount agreeing to the decimal. The trap is real, the maths is honest, and the picture and the simulation say the same thing.

06You are here

This chapter is the capstone of everything we have built about inference, so let's place it on the family tree. We added no new formula here. The base-rate trap is Chapter 6's flip, run in the rare-event regime: the prior is tiny, and the denominator fills with false positives. Everything upstream fed into it. The measure came from Chapter 3, conditioning from Chapter 5, and total probability and the flip from Chapter 6. All of it converges on the single most consequential mistake people make with probability in the real world. Find our rung, lit gold.

Now we turn one of the biggest corners in the whole book, so let's cross it deliberately. Look back at everything we have done since Chapter 1. It has all been about events: subsets of the sample space, things that either happen or don't. "The card is a spade." "The test came back positive." Yes or no, in or out. But notice how often we have secretly wanted a number instead: how many people test positive, how many heads in ten flips, how many of the crowd are truly sick. The instant you attach a number to each outcome, the event {k positives} stops being a yes/no subset and becomes the value k. That function from outcomes to numbers is the random variable, and the entire second half of this course is built on it. See the hinge.

That is the pivot. For seven chapters we have measured the chance that something happens. From here on we measure the chance that a number takes a particular value. That single shift unlocks distributions, expectation, variance, and the whole zoo of named laws waiting downstream. The base-rate trap was the last and hardest lesson of the events world. Now we give each outcome a number, and probability becomes something you can add, average, and predict. Turn the page.