Short-term Success, Long-term Fizzle
By Arnold Kling
Gordon Dahl and Lance Lochner write (I can’t find an ungated 2008 version of the paper, and the 2005 version seems to differ),
Our baseline estimates imply that a $1,000 increase in income raises combined math and reading test scores by 6% of a standard deviation in the short run. The gains are larger for children from disadvantaged families and are robust to a variety of alternative specifications. We find little evidence of long-run income effects, with most of the effects disappearing after one year.
In studies of interventions designed to improve education outcomes, a common result is to find statistically significant effects in the short run, but not in the long run. Researchers will report that “X improves test scores in 4th grade, but by 8th grade the difference has gone away.” Why is that? Possibilities:
1. Genetic factors are too strong. You can fool mother nature for a little while, but not for long. People tend to revert to their genetically-determined level of ability.
2. Other environmental factors are too strong. You can overcome environmental determinants for a little while, but nor for long. People tend to revert to their environmentally-determined level of skill development.
3. Over time, more random factors are introduced into the lives of the subject populations. Note, however, that this should lower the R-squared in the regression and might bring down the statistical significance of treatment effects, even if it does not cause the magnitude of treatment effects to disappear But researchers are fixated on statistical significance, so they interpret a drop in statistical significance as if it were a fall-off in the effect of the treatment.