Deadly Chicago Econometrics
By Arnold Kling
Bryan’s post on the statistical findings about parenting and health care reminds me of what we at MIT used to call “Chicago econometrics.”
A fundamental fallacy in classical statistics is to say, “Therefore, we accept the null hypothesis.” The classical statistical tests are designed to make it difficult to reject the null hypothesis (innocent in until proven guilty). See this lecture.
So, if you set up a classical test with the null hypothesis as “health care has no effect,” you are giving yourself a low probability of rejecting that hypothesis. Getting a statistically insignificant result is no more than that–an insignificant result. All claims that you have shown zero effect of the independent variable are fallacious, because you gave yourself a high probability of showing zero effect to begin with.
Why we called this “Chicago econometrics” I don’t know. Maybe it was just a way of venting at our rivals–kind of like those Maryland basketball fan T-shirts that say “Duck Fuke.”
In addition to the logical flaw, there is the practical problem that measurement error biases regression coefficients toward zero. So, if your measure of schooling, or parenting, or health care, is a poor proxy for the correct variable, then you will get a low coefficient for that reason alone.
One of my pet peeves is lazy econometrics, where someone tries to estimate an aggregate relationship for a disaggregated process. For example, you can show a significant relationship between many specific medical procedures and longevity for people with the relevant conditions. Yet if you take an aggregate proxy for medical care and an aggregate measure of longevity, there is no relationship. I think of the latter as lazy econometrics rather than a description of the real world.
Similarly, with parenting, my guess is that there are specific parenting practices that have significant effects on certain types of children. But trying to look at aggregate effects of “parents” on “children” while controlling for genetic effects gets you low coefficients due to lazy econometrics. Add to this the “Chicago econometrics” fallacy of saying “therefore we accept the null hypothesis, and you have a recipe for poor scientific practice.