Stat Fight, Con't
By Arnold Kling
a fair number of papers have a lot of data points. So rejecting the null hypothesis of no effect is pretty easy. Moreover, it is fairly common in both literatures to discuss the magnitude of the effect, not just statistical significance.
I agree with the technical point, but I would have to see specific papers. Sometimes, you do not have nearly as many data points as you appear to have. One of the reasons that I have given up hope of settling macroeconomic questions empirically is that with serial correlation and structural change, the effective number of data points is ridiculously small, regardless of what somebody might report as the “number of observations.” Those specific issues don’t affect the data that we are talking about, but other issues do.
He also says,
I agree with Arnold that we lose some valuable information when we look only at the aggregate effect of medicine or parenting. (Indeed, that was my whole point!) But that aggregate information is good to know, and one can imagine the public digesting it
All I can think of is the public digesting Paul Krugman’s argument that we spend more money on health care than countries with socialized medicine, we have the same longevity as those countries, therefore socialized medicine is more cost-effective.
In fact, we spend $2000 more per capita than other countries. Taking David Cutler’s value of a life-year as $100,000, the critical value for whether that $2000 is cost-effective is whether it increases our longevity by one week. (UPDATE: see the comments for a valid criticism of this arithmetic) If you want to try to find that week by controlling for all the other factors that affect longevity (genetics, homicide rates, traffic fatality rates, etc.), go ahead. But I think it’s like trying to call balls and strikes from a helicopter.
As to studies of parenting, I would be curious to know whether you as a pro-natalist regard your parenting style as being perfectly consistent across children. For each child, have you used the same method for getting the child to sleep through the night? For toilet training? Will each child have to wait until the same age to get a particular toy or a particular privilege?
If you are like most parents, your answer would be “no.” Yet aggregate studies of the effects of parenting almost always assume that the answer is “yes.”
And just when you want to say that “Even though I do some things differently with my children, I have the same basic personality with all of them,” someone like Judith Rich Harris will come along and say that the effect of your basic personality as a parent can’t be separated from your genetic effect.
I think that in both health care and parenting, there is a sleight-of-hand going on in claiming no effect. Intuitively, we think of the effect as big, based on a disaggregated view. But then the statistical “expert” comes along and does a macro study that introduces a lot of variation in the dependent variable due to extraneous factors, then picks a weak proxy for the factor under consideration, and pronounces the effect of that factor to be zero.