The Problem of False Positives
Mar. 9th, 2010 11:59 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
omorkaSo, there are two problems - they're AP Stats problems, but I give them to my IB Math Studies kids in the probability unit, too - that I give my kids every year.
---
Problem #1:
The researchers at Miskatonic University have developed a test for the dread disease migosis. One in 100 people is infected. The test has a 99% chance of delivering a correct diagnosis, and a 1% chance of delivering an incorrect one.
a) Suppose 10000 people are tested. List the expected values for the four groups: people who have migosis who are diagnosed with migosis; people who have migosis who are not diagnosed (false negatives); people who do not have migosis who are not diagnosed with it; and people who do not have migosis who are diagnosed with it (false positives).
b) Based on your results from part (a), how useful is this test for diagnosing migosis in the general population?
---
Problem #2.
Dr. Harkenbluff has invented a marvelous new machine that can detect students who have cheated on a test. The penalty at Illuminati University for cheating is immediate expulsion from the university. When placed on a student's head, the machine flashes a red light to indicate a cheater, and a green light to indicate an innocent student. However, the machine is not perfect. If it is placed upon the noggin of a vile cheater, there is a 1% chance that the machine fails to detect his or her perfidy, and instead flashes the green light. If, instead, it is placed upon the head of an honest student, there is a 0.5% chance that it detects some other impure thought and flashes the red bulb.
Dr. Harkenbluff' wishes to test every student on campus for cheating. His arch-nemisis, Professor Ravilaw, insists that the use of the device will result in too many innocent students being expelled.
a) Suppose that 50% of the student body are wicked cheaters. What is the probability that, given that the device has just flashed a red light, the student is guilty?
b) Suppose, instead, that only 0.1% of the student body engages in such immoral behavior. What is the probability that, given that the device has just flashed a red light, the student is guilty?
c) Who's right, Dr. Harkenbluff or Professor Ravilaw?
---
1) a) 100 people are expected to have migosis; 99 of them will be correctly diagnosed, 1 will be falsely cleared. 9,900 people do not have migosis; 9,801 of them will be correctly diagnosed, 99 will be falsely diagnosed with migosis.
b) Not very useful. Half of the people the test says have migosis are false positives.
2) a) P(C given R) = P (C and R)/P(R) = 0.495/0.4955 or about 99.9%
b) P(C given R) = P (C and R)/P(R) = 0.00099/0.005985 or about 16.5%
c) Professor Ravilaw. If the student body is largely honest, more innocent students will be expelled than cheaters. (Also accepted if (a) and (b) are correct: Professor Ravilaw, because even one innocent student expelled is too many.)
Now, why do I bring these up? There have been several topics that have appeared on my flist that involve the issue of catching a particular type of miscreant - criminals, terrorists, etc. Those who are in charge of apprehending such delinquents typically want to cast a wide net and test large numbers of people - say, drug-testing everyone in a large corporation or scanning everyone who passes through an airport.
As long as the problems are rare and the testing method is imperfect, this is likely to fail, because of the problem of false positives.
Suppose we take the example of a terrorist in a subway station that thousands of people pass through every day, and an explosive-sniffing dog. Even if the dog is very good, if it has any error rate at all, the majority of the people it triggers on - probably the vast majority, since I suspect the probability of any given subway rider being a terrorist is significantly less than 0.5% - will not, in fact, be terrorists. Day after day, the dog will be crying wolf, through no fault of its own. The handlers will get complacent, the inspectors lazy, and the day the dog triggers on the guy with the C4 in his backpack, they'll be caught nearly as flatfooted as they would have been without the dog.
(An aside on the airport example: suppose a, gosh, what hasn't been used yet, a t-shirt bomber is in the scanning line and gets detected. What's to stop them from detonating right there in the airport? Isn't that the logical thing for a suicide bomber to do? And isn't the property and human damage likely to be roughly the same as if they'd taken out a plane? So how exactly does the scanning protect anyone? It just changes the location of the explosion, doesn't it?)
As long as what you're looking for is rare enough, you'll go on more wild goose chases than anything else, and if your manpower is limited, you'll almost certainly be chasing geese when the fox shows up. :-/
Someone, somewhere, failed to Do The Math.
---
Problem #1:
The researchers at Miskatonic University have developed a test for the dread disease migosis. One in 100 people is infected. The test has a 99% chance of delivering a correct diagnosis, and a 1% chance of delivering an incorrect one.
a) Suppose 10000 people are tested. List the expected values for the four groups: people who have migosis who are diagnosed with migosis; people who have migosis who are not diagnosed (false negatives); people who do not have migosis who are not diagnosed with it; and people who do not have migosis who are diagnosed with it (false positives).
b) Based on your results from part (a), how useful is this test for diagnosing migosis in the general population?
---
Problem #2.
Dr. Harkenbluff has invented a marvelous new machine that can detect students who have cheated on a test. The penalty at Illuminati University for cheating is immediate expulsion from the university. When placed on a student's head, the machine flashes a red light to indicate a cheater, and a green light to indicate an innocent student. However, the machine is not perfect. If it is placed upon the noggin of a vile cheater, there is a 1% chance that the machine fails to detect his or her perfidy, and instead flashes the green light. If, instead, it is placed upon the head of an honest student, there is a 0.5% chance that it detects some other impure thought and flashes the red bulb.
Dr. Harkenbluff' wishes to test every student on campus for cheating. His arch-nemisis, Professor Ravilaw, insists that the use of the device will result in too many innocent students being expelled.
a) Suppose that 50% of the student body are wicked cheaters. What is the probability that, given that the device has just flashed a red light, the student is guilty?
b) Suppose, instead, that only 0.1% of the student body engages in such immoral behavior. What is the probability that, given that the device has just flashed a red light, the student is guilty?
c) Who's right, Dr. Harkenbluff or Professor Ravilaw?
---
1) a) 100 people are expected to have migosis; 99 of them will be correctly diagnosed, 1 will be falsely cleared. 9,900 people do not have migosis; 9,801 of them will be correctly diagnosed, 99 will be falsely diagnosed with migosis.
b) Not very useful. Half of the people the test says have migosis are false positives.
2) a) P(C given R) = P (C and R)/P(R) = 0.495/0.4955 or about 99.9%
b) P(C given R) = P (C and R)/P(R) = 0.00099/0.005985 or about 16.5%
c) Professor Ravilaw. If the student body is largely honest, more innocent students will be expelled than cheaters. (Also accepted if (a) and (b) are correct: Professor Ravilaw, because even one innocent student expelled is too many.)
Now, why do I bring these up? There have been several topics that have appeared on my flist that involve the issue of catching a particular type of miscreant - criminals, terrorists, etc. Those who are in charge of apprehending such delinquents typically want to cast a wide net and test large numbers of people - say, drug-testing everyone in a large corporation or scanning everyone who passes through an airport.
As long as the problems are rare and the testing method is imperfect, this is likely to fail, because of the problem of false positives.
Suppose we take the example of a terrorist in a subway station that thousands of people pass through every day, and an explosive-sniffing dog. Even if the dog is very good, if it has any error rate at all, the majority of the people it triggers on - probably the vast majority, since I suspect the probability of any given subway rider being a terrorist is significantly less than 0.5% - will not, in fact, be terrorists. Day after day, the dog will be crying wolf, through no fault of its own. The handlers will get complacent, the inspectors lazy, and the day the dog triggers on the guy with the C4 in his backpack, they'll be caught nearly as flatfooted as they would have been without the dog.
(An aside on the airport example: suppose a, gosh, what hasn't been used yet, a t-shirt bomber is in the scanning line and gets detected. What's to stop them from detonating right there in the airport? Isn't that the logical thing for a suicide bomber to do? And isn't the property and human damage likely to be roughly the same as if they'd taken out a plane? So how exactly does the scanning protect anyone? It just changes the location of the explosion, doesn't it?)
As long as what you're looking for is rare enough, you'll go on more wild goose chases than anything else, and if your manpower is limited, you'll almost certainly be chasing geese when the fox shows up. :-/
Someone, somewhere, failed to Do The Math.
