Strongly recommended. A sample:
within this universe of programs that are far more likely to fail than succeed, programs that try to change people are even more likely to fail than those that try to change incentives. A litany of program ideas designed to push welfare recipients into the workforce failed when tested in those randomized experiments of the welfare-reform era; only adding mandatory work requirements succeeded in moving people from welfare to work in a humane fashion. And mandatory work-requirement programs that emphasize just getting a job are far more effective than those that emphasize skills-building. Similarly, the list of failed attempts to change people to make them less likely to commit crimes is almost endless–prisoner counseling, transitional aid to prisoners, intensive probation, juvenile boot camps–but the only program concept that tentatively demonstrated reductions in crime rates in replicated RFTs was nuisance abatement, which changes the environment in which criminals operate.
Read the whole thing. If David Brooks is going to give out his annual awards for most important essays, I would nominate this one.
One of the lessons that is implicit in the essay (and that I think that Manzi ought to make explicit) is, “Don’t trust one-offs.” That is, do not draw strong conclusions based on a single experiment, no matter how well constructed. Instead, wait until many experiments have been conducted, in a variety of settings and using a variety of techniques. An example of a one-off that generated a lot of recent excitement is the $320,000 kindergarten teacher study.
READER COMMENTS
Patri Friedman
Aug 2 2010 at 10:22pm
Change incentives, not people…i think that can be applied to politics somehow too :).
Contemplationist
Aug 2 2010 at 11:08pm
Did the kindergarten teacher study control for kids’ IQs? If not, you’d do better to read Steve Sailer than this hogwash. It doesn’t cost billion dollars either
TTE
Aug 2 2010 at 11:32pm
Very interesting albeit depressing read. Quantum physics is a breeze compared to predicting the effect of a change in the price of apples.
The description of how markets improved on the experimentation process was fascinating. The power and evolutionary capacity of markets never fail to amaze me.
Robert Bell
Aug 3 2010 at 12:13am
“If David Brooks is going to give out his annual awards for most important essays, I would nominate this one”
Alternatively, the NYT could just give his slot to Jim Manzi …
Steve Sailer
Aug 3 2010 at 3:25am
The social sciences have actually uncovered a lot of highly useful knowledge. The problem is that what the social sciences have demonstrated to be true tends not to be popular with social scientists.
Floccina
Aug 3 2010 at 9:56am
The fact that so much study goes against the above and that it is hard to think of a mechanism that would make K teacher benefits jump a few years and the that the above is what they would like to find makes me discount its findings to close to zero. I think that they must have made some kind of mistake or somehow let their biases in.
People are desperate to close the racial achievement gap and so seem to be moving toward getting black children away from their parents and into the hands of trained Gov employees earlier. The Gov employees may be more capable but for the most part they do not love the children more than the parents do.
eccdogg
Aug 3 2010 at 11:22am
Great article.
In my opinion this type of thinking is the strongest argument for the classical conservative position.
Rules and institutions that have been established based on trial and error of billions of people over thousands of years should not be discarded wlly nilly and we should be sceptical of all grand social plans (including libertarianism even though I am a libertarian)
One huge thing I have learned in my business life is the huge value of making small experiments and learning by doing. Then readjusting based on what you learned and then taking anothe incremental step.
Thucydides
Aug 3 2010 at 1:37pm
Steve Sailer’s point that the stuff found to be true by social science is not to the social scientists’ liking, is borne out by Manzi’s point that programs that seek to change people don’t work, while those that change incentives may be somewhat effective.
Social scientists tend to hold left-liberal views, and the belief in human perfectibility is central to that ideology. Probably that is why they chose the field. The idea that it is usually futile to try to change people is not going to be very welcome with them.
fundamentalist
Aug 3 2010 at 1:45pm
What Manzi fails to tell readers, probably because he doesn’t know, is that controlled experiments has not always been the sole criteria for the social sciences. In the early days of social science, logic was the preferred method of advancement. Social sciences used observation and logic. Geometry was the model. Along about 1900 social scientists decided they needed to be like the natural sciences in every way. The shift was pure ideology. There was no philosophical, logical or empirical reason for the shift. Some argue that the shift came about because socialists were unable to defend themselves against the logical onslaught of free market economists, so they promoted the controlled experiment as the only path to science. For a better explanation, read Hayek’s “The Counter-Revolution in Science” and Mises’ “Human Action”, the first section.
Colin K
Aug 3 2010 at 2:27pm
I love this line from the article:
In other words, they have a bunch of data, and a bunch of guesses. As Homer (of Simpson) says, “Woohoo!”
Also, 12,000 people in one Tennessee town sounds like an awfully small sample about which to conclude anything that’s supposed to apply outside that particular area code.
Chris T
Aug 3 2010 at 3:38pm
Social science went awry when it thought of human behavior as separate from, rather than emergent from, biology.
Thus we get all these ‘experiments’ and ‘studies’ that pretend there’s no such thing as genetics.
Chris Koresko
Aug 3 2010 at 3:58pm
Very nice article. It dovetails nicely with your Doubtbook, too. Maybe you would consider expanding its scope to present the thesis that aside from a few specific areas in hard science, we really don’t understand much of anything about ourselves or the world around us.
I’ve long believed that our civilization went awry by adopting the attitude that the ability to predict the motions of the planets meant that every phenomenon was quantifiable, predictable, and understandable. Hence “scientific” theories of economics and society displaced learned and revealed knowledge. It seems we’re only now becoming aware as a society how big a mistake that was.
Jeremy, Alabama
Aug 3 2010 at 4:35pm
Excellent article, thanks for linking.
I am reminded of a Richard Feynman’s story about rat-running experiments (a short version is in his wikipedia entry).
Chris T
Aug 3 2010 at 7:11pm
Chris K – Human behavior can be understood to some degree, the problem is that’s not what social science originally set out to do. SS’s original (and current for far too many) goal was to perfect humanity, not understand it.
This necessitates specific beliefs about humanity which tend to be inimical to understanding it.
fundamentalist
Aug 4 2010 at 9:13am
Excellent points Chris T!
Brian Clendinen
Aug 4 2010 at 10:15am
@Floccina
Actually this would have the totally opposite effect. A Brookings institution study (I wish I could find a link to), found it was single mothers (not divorced or widowed) that explained 100% of the racial and income achievement gaps among kids.
Chris T, great point that SS’s founding principals are more religious than scientific.
Comments are closed.