Predictability and the Base Rate

Whenever a statistician wants to predict the likelihood of some event based on the available evidence, there are two main sources of information that have to be taken into account: (1) the evidence itself, for which a reliability figure has to be calculated; and (2) the likelihood of the event calculated purely in terms of relative incidence. The second figure here is the base rate. Since it is just a number, obtained by the seemingly dull process of counting, it frequently gets overlooked when there is new information, particularly if that new information is obtained by “experts” using expensive equipment. In cases where the event is dramatic and scary, like a terrorist attack on an airplane, failure to take account of the base rate can result in wasting massive amounts of effort and money trying to prevent something that is very unlikely.

For example, suppose that you undergo a medical test for a relatively rare cancer. The cancer has an incidence of 1 percent among the general population. (That is the base rate.) Extensive trials have shown that the reliability of the test is 79 percent. More precisely, although the test does not fail to detect the cancer when it is present, it gives a positive result in 21 percent of the cases where no cancer is present—what is known as a false positive. When you are tested, the test produces a positive diagnosis. The question is: What is the probability that you have the cancer?

If you’re like most people, you’ll assume that if the test has a reliability rate of nearly 80 percent, and you test positive, then the likelihood that you do indeed have the cancer is about 80 percent (i.e., the probability is approximately 0.8). Are you right?

The answer is no. You have focused on the test and its reliability and overlooked the base rate. Given the scenario just described, the likelihood that you have the cancer is a mere 4.6 percent (i.e., 0.046). That’s right—there is a less than 5 percent chance that you have the cancer. Still a worrying possibility, of course. But hardly the scary 80 percent you thought at first.

Keith Devlin explains why the accuracy of tests and measurments must take into account the base rate for the phenomenon.