Data and Dogma
By Arnold Kling
(i). Bob Sutton quotes a meta-analysis by Frank Schmidt and John Hunter published in 1998 on the factors found to affect job performance.
1. GMA tests (“General mental ability”)
2. Work sample tests
3. Integrity tests: surveys design to assess honesty … I don’t like them but they do appear to work.
4. Conscientiousness tests…
12. Job experience (years)…
16. Years of education
18. Graphology (e.g., handwriting analysis)
I love it that “years of education” just barely beats out handwriting analysis.
Jackson’s findings showed that outside of flu season, the baseline risk of death among people who did not get vaccinated was approximately 60 percent higher than among those who did, lending support to the hypothesis that on average, healthy people chose to get the vaccine, while the “frail elderly” didn’t or couldn’t. In fact, the healthy-user effect explained the entire benefit that other researchers were attributing to flu vaccine, suggesting that the vaccine itself might not reduce mortality at all.
A basic lesson in AP statistics concerns the differences between an observational study and an experiment. There is a well-known finding that people who get flu vaccinations have lower death rates than people who do not. But this finding is not based on an experiment. It is instead based on observation of people who choose to get shots and people who do not. My hypothesis is that people who get flu shots are more conscientious than people who do not, and more conscientious people have lower death rates. Whatever the reason, the article cites research where what economists would call “natural experiments” show that flu shots do not affect death rates.
Consider the well-known observation that people with health insurance have lower death rates than people without health insurance. This, too, is based on an observational study, and once again it is likely that conscientiousness is a confounding factor. The Rand health study, which was a controlled experiment, found no differences in outcomes from different levels of insurance.
A final point is that in the public debate, I suspect that the demand is not for reliable studies but for studies that support one’s point of view. Brownlee and Lenzer report on the way that vaccine skeptics are bullied by those with whom they disagree.
My advice to laymen would be to pay attention to how scientists argue. When the ad hominem attacks and bullying are flowing freely, assume that the attacking side has problems assembling convincing evidence.