The Problem of False Positives

[omorka]

So, there are two problems - they're AP Stats problems, but I give them to my IB Math Studies kids in the probability unit, too - that I give my kids every year.

---

Problem #1:

The researchers at Miskatonic University have developed a test for the dread disease migosis. One in 100 people is infected. The test has a 99% chance of delivering a correct diagnosis, and a 1% chance of delivering an incorrect one.

a) Suppose 10000 people are tested. List the expected values for the four groups: people who have migosis who are diagnosed with migosis; people who have migosis who are not diagnosed (false negatives); people who do not have migosis who are not diagnosed with it; and people who do not have migosis who are diagnosed with it (false positives).

b) Based on your results from part (a), how useful is this test for diagnosing migosis in the general population?

---

Problem #2.

Dr. Harkenbluff has invented a marvelous new machine that can detect students who have cheated on a test. The penalty at Illuminati University for cheating is immediate expulsion from the university. When placed on a student's head, the machine flashes a red light to indicate a cheater, and a green light to indicate an innocent student. However, the machine is not perfect. If it is placed upon the noggin of a vile cheater, there is a 1% chance that the machine fails to detect his or her perfidy, and instead flashes the green light. If, instead, it is placed upon the head of an honest student, there is a 0.5% chance that it detects some other impure thought and flashes the red bulb.

Dr. Harkenbluff' wishes to test every student on campus for cheating. His arch-nemisis, Professor Ravilaw, insists that the use of the device will result in too many innocent students being expelled.

a) Suppose that 50% of the student body are wicked cheaters. What is the probability that, given that the device has just flashed a red light, the student is guilty?

b) Suppose, instead, that only 0.1% of the student body engages in such immoral behavior. What is the probability that, given that the device has just flashed a red light, the student is guilty?

c) Who's right, Dr. Harkenbluff or Professor Ravilaw?

---


1) a) 100 people are expected to have migosis; 99 of them will be correctly diagnosed, 1 will be falsely cleared. 9,900 people do not have migosis; 9,801 of them will be correctly diagnosed, 99 will be falsely diagnosed with migosis.

b) Not very useful. Half of the people the test says have migosis are false positives.


2) a) P(C given R) = P (C and R)/P(R) = 0.495/0.4955 or about 99.9%

b) P(C given R) = P (C and R)/P(R) = 0.00099/0.005985 or about 16.5%

c) Professor Ravilaw. If the student body is largely honest, more innocent students will be expelled than cheaters. (Also accepted if (a) and (b) are correct: Professor Ravilaw, because even one innocent student expelled is too many.)


Now, why do I bring these up? There have been several topics that have appeared on my flist that involve the issue of catching a particular type of miscreant - criminals, terrorists, etc. Those who are in charge of apprehending such delinquents typically want to cast a wide net and test large numbers of people - say, drug-testing everyone in a large corporation or scanning everyone who passes through an airport.

As long as the problems are rare and the testing method is imperfect, this is likely to fail, because of the problem of false positives.

Suppose we take the example of a terrorist in a subway station that thousands of people pass through every day, and an explosive-sniffing dog. Even if the dog is very good, if it has any error rate at all, the majority of the people it triggers on - probably the vast majority, since I suspect the probability of any given subway rider being a terrorist is significantly less than 0.5% - will not, in fact, be terrorists. Day after day, the dog will be crying wolf, through no fault of its own. The handlers will get complacent, the inspectors lazy, and the day the dog triggers on the guy with the C4 in his backpack, they'll be caught nearly as flatfooted as they would have been without the dog.

(An aside on the airport example: suppose a, gosh, what hasn't been used yet, a t-shirt bomber is in the scanning line and gets detected. What's to stop them from detonating right there in the airport? Isn't that the logical thing for a suicide bomber to do? And isn't the property and human damage likely to be roughly the same as if they'd taken out a plane? So how exactly does the scanning protect anyone? It just changes the location of the explosion, doesn't it?)

As long as what you're looking for is rare enough, you'll go on more wild goose chases than anything else, and if your manpower is limited, you'll almost certainly be chasing geese when the fox shows up. :-/

Someone, somewhere, failed to Do The Math.

Comments

[yermie.livejournal.com]

While I agree with your conclusions, the simple answer is to retest.

Taking Problem 1 into account:

Of your 10k sample, 9801 of them are clean. Remove those.

Test the remaining 199 again.

This time, your percentages for Diseased are off (somewhere close to 50/50), but the test accuracy isn't. So 99% of the 199 will be properly diagnosed, leaving only the 1% "double failures". Which brings your number of misdiagnosed people down to 2.

Eventually, you have to trust the odds to show that you're getting a valid sample.

As for your "real world application", if the drug / bomb dog picked out at a 0.5% error per pass, then a second / third pass should clear 99.5% of the false positives.

But, yes. If I were a suicide bomber, trying to get on a plane with a bomb / gun / knife / insert_hazardous_item_here, taking out the security screening area (along with passengers and guards) might be a valid set of targets. Unless you were planning on using said item to gain cockpit access (or blow up near something, and hope to crash via "What goes up..."), and crash the plane into something else. Of course, once you've been "made", you may as well blow it then, and take out what you can.

(Note to FBI/ NSA / CIA / DHS / etc.: I'm not condoning such behavior. Just check my wiretaps, you'll see...)

[quantumduck.livejournal.com]

I figure all the TSA posturing has more to do with the 'theatre of security' than security itself. Accuracy is less important than the psychology.

Those of us who are sufficiently familiar with the technology are aware that polygraph tests are easily outwitted. The whole point of such tests is to create a convenient fiction that authorities CAN actually correctly identify misbehavior.

In my experience misbehavior and misapplication of such tests (by authorities) far outweigh the misbehavior of the population being tested in the first place.

[panthyr.livejournal.com]

Well, not quite. Given the 1% false negative, if you remove the cleared people, you are taking 9,802 out of the sample - 1 of whom has migosis. Depending on the fatality and transmission rate, that may be one too many to prevent the migosis apocalypse. :) The solution, then, is the same as you applied to the second problem, which is to test everyone again. At that point, however, you need to start questioning the factors that produce false results. Does the test react wrongly to anyone who has eaten a banana (or had sex with a Deep One) in the last 48 hours? Will the same people get false results?
Re: security theater, both of the "bombers" that got through had Rube Goldberg-ish explosives that required a certain uninterrupted amount of time to properly deploy. Had they been stopped at the gate, they would have been unable to set their shoes/crotch on fire. At this point, a true bomb at the security checkpoint would be a lot more effective.

[fiona-conn.livejournal.com]

Is it worth pointing out that if there's 1 in 100 people infected, that that means out of 100 people, 99 aren't, and since the test has 1% failure rate, that it's entirely possible for that 1 out of 100 to be a false positive?

[fiona-conn.livejournal.com]

Note: I didn't read the solution before posting this! xD

[fiona-conn.livejournal.com]

I would have to agree -- the psychology of fear vs security can be pretty powerful.

Yesterday, when I was out taking photos in Glasgow, I got stopped by a security guard at a shopping mall. I wasn't doing anything wrong, I was just trying to work out the best angle to get a good picture from, 'cause of the way the corridor curves and I wanted to capture that. But the guard told me I wasn't allowed to take photos because it posed a security risk, and went on to explain that they get foreigners who come and take photos as well. Essentially telling me that stopping me was to do with somehow stopping terrorism.

I was pretty much bemused by the whole thing. I've seen these kinds of things talked about before, but, it's the first time I've ever had that happen.

[omorka.livejournal.com]

The previous comment already caught the false negative and lurking variables, so I'll just add: this works reasonably well if testing is short, non-invasive, non-destructive, and cheap, and if the condition you're testing for is not so rare that the event described is the beginning of the second round of testing.

If the original event is not one in 10,000 but 1 in 1,000,000, this is Round Two already, and you have to go to Round Three. And it's worse if your test has a failure rate worse than 1%.

A great deal of testing is not short, not non-invasive, and damned expensive. The costs of double- and triple-testing quickly get prohibitive.