James Manzi on Experimental Social Science
Strongly recommended. A sample:
within this universe of programs that are far more likely to fail than succeed, programs that try to change people are even more likely to fail than those that try to change incentives. A litany of program ideas designed to push welfare recipients into the workforce failed when tested in those randomized experiments of the welfare-reform era; only adding mandatory work requirements succeeded in moving people from welfare to work in a humane fashion. And mandatory work-requirement programs that emphasize just getting a job are far more effective than those that emphasize skills-building. Similarly, the list of failed attempts to change people to make them less likely to commit crimes is almost endless–prisoner counseling, transitional aid to prisoners, intensive probation, juvenile boot camps–but the only program concept that tentatively demonstrated reductions in crime rates in replicated RFTs was nuisance abatement, which changes the environment in which criminals operate.
Read the whole thing. If David Brooks is going to give out his annual awards for most important essays, I would nominate this one.
One of the lessons that is implicit in the essay (and that I think that Manzi ought to make explicit) is, “Don’t trust one-offs.” That is, do not draw strong conclusions based on a single experiment, no matter how well constructed. Instead, wait until many experiments have been conducted, in a variety of settings and using a variety of techniques. An example of a one-off that generated a lot of recent excitement is the $320,000 kindergarten teacher study.