What’s the empirical evidence that RCTs actually improve policy? Incisive comments from the noble Lant Pritchett:
The argument that RCTs would be more than a tiny component of the overall process of improving development outcomes seems, even now, 10 years in, at best not provable and at worst not very likely—as it is at odds with some basic known facts about development and about policy formulation and implementation.
What “basic known facts” does he have in mind? First, RCTs can’t be crucial for development, because many countries became highly developed long before RCTs mattered.
First of all, the argument that RCTs had, until recently, been used sparingly, if at all, and yet are important in achieving good outcomes sits in kind of embarrassing counterpoint with the obvious fact that lots of countries have really good outcomes. That is whether one uses the Human Development Index or the OECD Better Life Index or any social indicator—from poverty to education to health to life satisfaction—there is a similar set of countries near the top… No one has ever made the arguments that these countries are developed and prosperous because they used rigorous evidence—much less RCTs—in formulating policy and programs. While one might have faith that RCTs can help along the path to development, RCTs didn’t help for those that are there now.
Second, the highest observed economic growth in human history happened in the decades right before RCTs became fashionable:
Second, at about the time (early 2000s) that the randomista movement, which often claimed to be about reducing poverty, was gaining steam, several countries were experiencing or had experienced the rapidest reductions in low-bar absolute poverty in the history of man. Indonesia, China, Vietnam had all seen dollar-a-day–like poverty measures fall from well over half of their population to under 20 percent in less than 30 years. In Vietnam the World Bank figures show dollar-a-day poverty falling from over 40 percent to 16.9 percent just between 2002 and 2008. If these countries did this completely without any use of RCTs (and they did), then from where exactly does the argument that using RCTs will accelerate poverty reduction come? Is there any evidence that countries that used more RCTs had more rapid poverty reduction than those that didn’t? No. While it might be the case that RCTs could accelerate poverty reduction this was (and is) a faith-based, not evidence-based, claim.
Third, the past is the best predictor of the future – and past RCTs had little influence on policy:
Third, there was the recent historical experience with RCTs in social policy in the USA. The randomistas were not proposing new methods or techniques but rather broader adoption into the field of development methods that already had a long history. There was a big fad toward the use of experiments in a variety of social policy domains in the USA in the 1970s. The Rand Health Insurance Experiment, carried out from 1971 to 1986, is still the largest health-policy experiment ever done. Income maintenance (negative income tax) experiments were carried out between 1968 and 1982 in four sites around the USA. A 1970 act of Congress authorized experiments such as the Housing Allowance Demand Experiment. The Kansas City Police Preventive Patrol experiment was carried out in 1972–73. This created ample capacity in the USA for doing social experiments inside organizations like Rand, Abt Associates, and Mathematica Policy Research, among others. At the founding of J-PAL there was 35 years of experience with RCTs in the USA on which to draw evidence of their efficacy (or not) as tools for improving policy and hence human outcomes. You would have thought that if the widespread use of RCTs had strongly impacted policy in the USA, this would have been put forward as evidence. Strangely, whether or not decades of social policy RCTs actually did have impact on policies and outcomes in the USA just kind of never came up in arguing that they would in developing countries.
What’s the alternative? Pritchett elsewhere argues that orthodox growth economics worked wonders for humanity but got almost no credit. In fact, it got turned into the bizarre conspiracy theory of “neoliberalism” and denounced by left and right alike. Fortunately, as Vanilla Sky teaches us, “Every passing minute is another chance to turn it all around.” Growth economists have made plenty of mistakes, but they should admit that RCTs are a trendy distraction and pick up where they left off.
READER COMMENTS
Nathan Barnard
Feb 24 2022 at 10:17am
This seems like a strange argument. It seems likely that the countries which are currently poorest and growing the least are the ones where past methods haven’t been sufficient to engender growth and so we should try new tools. It would also be pretty great if we could find a way of making industrial revolutions less terrible for the people living through them.
The argument I was expecting was that the poorest countries have the worst governance and then maybe some empirical evidence about lack of policy changes based on RCT results.
Thomas Strenge
Feb 24 2022 at 3:53pm
What’s RCT?
Alan Goldhammer
Feb 24 2022 at 4:03pm
Randomized Controlled Trial. They are most commonly used in the testing of new drugs for regulatory approval. Their use in the social sciences is less obvious.
David Killion
Feb 24 2022 at 6:23pm
Randomized Control Trial
Thomas Strenge
Feb 25 2022 at 8:31am
Thanks!
Thomas Lee Hutcheson
Feb 24 2022 at 4:28pm
RTC are ok as a way of better distributing the tiny amounts of development aid, but development results rom removing the obstacles to productive private investment supported by public investment with positive NPV’s
TGGP
Feb 25 2022 at 9:35am
RCTs typically give results in line with the Iron Law of Evaluation: “the expected value of any net impact assessment of any large scale social program is zero”. So it mostly tells people that their ideas don’t work, which they don’t like hearing and are inclined to just ignore.
Matthias
Feb 25 2022 at 10:15pm
I’d like to see more RCTs at the margins.
Not necessarily for spending much money or big policy question. Just stuff like: start some high schools 5 minutes early and some 5 minutes later. See whether you get some effects.
Assume starting slightly later proves beneficial; then move more schools to that later start time, and restart experiments from the new base line.
These mini RCTs would also be useful exactly because of the iron law of evaluation:
Most interventions only have a minuscule or even zero effect. RCTs would show exactly that, and thus see fewer interventions deployed on a large scale.
Doing lots of RCTs as a matter of course might also encourage authorities to find ways to make policies variable on a local level. That’s good for subsidiarity. Subsidiarity taken to its logical extreme is almost as good as libertarianism.
Comments are closed.