Ian Ayres’ Super Crunchers is full of neat material. But my favorite parts highlight the innumeracy of the medical profession. Most vivid example:
[M]y partner, Jennifer, and I were expecting for the first time – back in 1994. Back then, women were told the probability of Down symdrome based on their age. After sixteen weeks, the mother could have a blood test… and then they’d give you another probability. I remember asking the doctor if they had a way of combining the different probabilities. He told me flat out, “That’s impossible. You just can’t combine probabilities like that.”
Can’t med schools cut one lecture on anatomy to make room for Eliezer’s punchy essay on Bayesian reasoning?
Until they do, of course, you could defend doctors by saying “That isn’t taught in medical school.” So let’s move on to two stories where medical professionals make errors that should embarrass a sixth-grader.
The first involves Ayres’ friend Ben Polak:
“I had a comical interaction with a very nice physician… where she said, ‘One of these probabilities is 1 in 1,000, and one is 1 in 10,000, so what’s the difference?’ and to me there’s actually quite a big difference. There’s a ten times difference.”
The other involves Ayres himself:
I had a similar interaction with a genetic counselor… When I asked for a probability of Down syndrome, the counselor unhelpfully offered: “That would only be a number. In reality, your child will either have it or not.”
In my experience, doctors exacerbate their innumeracy with an idiotic “You can’t be too careful” mantra. It’s almost like something out of a Robin Hanson model.
Actually, it’s exactly like that. Argh.
READER COMMENTS
John Thacker
Jan 20 2008 at 7:40pm
In fairness to the first excerpt, I can imagine a reasonable meaning for “You just can’t combine probabilities like that.” It is indeed possible that the two probabilities are not independent, and that medical science may not have yet investigated the correlation between positive tests on each. It is entirely reasonable to warning people against assuming the the probabilities are independent and “just combin[ing] probabilities like that.” Of course, that may not be what he meant…
It’s not impossible, of course, to combine the probabilities but it’s likely that the medical profession may not know what the best estimate is in the universe of possibilities because they don’t know the conditional probability. The two tests might be essentially measuring the same thing, in which case they would turn up similar false positives and false negatives, they might be independent, or they might even have superb resolution because one discriminates well against the false positives of the other.
The essay on Bayes’s Theorem is important, but it’s not sufficient for understanding the problem of how the probability changes with two consecutive tests instead of one.
The other excerpts are basically indefensible, certainly.
John Thacker
Jan 20 2008 at 7:42pm
Of course, in the example given, I suspect that the reasons for Down’s syndrome being more likely at a certain age is roughly independent of what a blood test would pick up, so the simple Bayes’s Theorem application would be relevant.
Steve
Jan 21 2008 at 1:16am
I’m pretty sure a statistics course including Bayes theorem is part of the required curriculum for accreditation in US medical schools. It was when I went through a little over 10 years ago. The problem is you don’t really have a need to sit down and crunch out a T+/D+ Bayesian 4 square table on a daily basis, and it quickly fades from memory. What remains is a vague recollection of ideas like the fact that a medical test is pretty worthless if the prevalence of a disease in a population is pretty low.
I’m a bit confused with the first example. It’s been a while since I’ve done obstetrics, but when we used to order the blood test they’re talking about, it would include the maternal age in calculating the probability of the baby having Down’s. The probabilities have already been “combined” when you get the test results back.
Regarding the second example (1 in 1000 vs 1 in 10k) I’m guessing here since I haven’t read the book, but I think the nice Dr. was politely trying to say “either way, it’s pretty damn unlikely, so go home and quit worrying about it.”
In the third example the only excuse I can see is that they ran into a real touchy-feely type genetic counselor rather than a numbers-cruncher. I sort of see the counselor’s point – when the baby comes out it either has Down’s or it doesn’t. But there’s not much point in meeting with a genetic counselor if they can’t give you some numbers.
The reality with medicine is not that the numbers an probabilities don’t matter, it’s just that they often take second seat to other issues. Google up “failure to diagnose cancer” for an example.
Jerome
Jan 21 2008 at 5:37am
Actually, it is a standard procedure to combine several Down syndrom screens to get a more accurate result. Commong procedures integrate three to four screens. Just look at the “Down syndrome” entry on Wikipedia.
Dan Weber
Jan 21 2008 at 8:38am
Actually, it’s a difference of .0009, or .09%.
While saying “this raises your risk of foot cancer 100 times!” sounds scary, and it makes for easy quotes in the newspaper, if the risk of foot cancer was only 1-in-a-million to start with, your total risk has gone up a negligible amount.
(If this is something you’re going to do a thousand times over the course of your life, going from 1:10,000 to 1:1000 is significant.)
Chuck
Jan 21 2008 at 10:20am
I have to agree that, for a single sample (like a person doing something one time), going from 1 in 1000 to 1 in 10000 is a pretty meaningless difference.
Furthermore, I wonder how much of the ‘over medicating’ that doctors do is not actually driven by fear of lawsuits, but rather feeling responsible for any patient of theirs that dies, wether or not they could reasonably have prevented it.
John Thacker
Jan 21 2008 at 11:40am
Actually, it is a standard procedure to combine several Down syndrom screens to get a more accurate result. Commong procedures integrate three to four screens. Just look at the “Down syndrome” entry on Wikipedia.
Certainly it is, but that’s because there’s sufficient research on those particular combinations to determine the joint probabilities. One could not, for example, run the “Triple screen” followed by the “Quad screen” and simply expect to assume that the false positive rates were independent. Three of the four tests in the quad screen are the three used in the triple screen.
I don’t see similar data for combining the quad screen with amniocentesis. It’s likely that amniocentesis is so accurate that there is little concern for exactly how independent it is of the initial screens. After all, even in the worst case, when one combines tests one does have a statistical power equal to at least the stronger of the two tests.
Imagine something like picking 4 lottery numbers from 0 to 9. Your chance of having the winning set of numbers is 1 in 10000. A “test” that examines the first number may come back positive, raising one’s chance to 1 in 1000. (We’ll even ignore errors for now.) A further test that examines the first two numbers, if it comes back positive, raises the chance to 1 in 100 whether the first test was performed or not. OTOH, if the second test examines the last two numbers, a positive result does combine to raise the chance to 1 in 10 if both are performed, even though the power of the second test by itself is unchanged. There are all sorts of intermediate tests with the same power by themselves, but with different properties when combined.
There is something to be said for not attempting to blindly combine the statistical power of two tests without knowing about the conditional probabilities involved. Merely using the results of the more powerful test, as the physician indicated, does put a bound on the possible result. I don’t doubt that in many cases more is known about the joint probabilities and assumptions of greater independence are justified, but in some cases counseling against crude assumptions of independence is understandable.
Comments are closed.