In economics, Randomized Controlled Trials (RCTs) now stand at the pinnacle of the methodological hierarchy. “Natural experiments” are a distant second. Work based on old-fashioned observational data is actually hard to publish anywhere prestigious. For many scholars, RCTs aren’t just the gold standard of research. Nothing else is even fungible. This is the Age of the Randomista.
Which raises a serious problem: How can researchers address questions where no RCT is feasible? To do an RCT on national monetary policy, for example, you would have to randomly assign monetary policies to a bunch of countries. Not gonna happen. To do an RCT on national disincentive effects of welfare, you would have to randomly assign welfare policies to a bunch of countries. Again, not gonna happen.
Sure, you could run some RCT laboratory experiments on monetary policy. But why assume that some silly games in a lab carry over into the real world? Similarly, you could run a pilot welfare program for a single city and measure the effects. But perhaps a lot of the labor supply response comes from the society-wide erosion of stigma against idleness. If so, your pilot program will fail to detect it.
The same goes if a Marxist claims that once capitalism has been eliminated, people will work for the sheer joy of contributing to the community. You could try running an experiment on a utopian commune, but the Marxist could protest, “Capitalism must be eliminated world-wide before my claim holds.” And on the flip side, the collapse of Communism wasn’t based on RCTs either. Critics just said, “This is an awful system and must be dismantled.” And amazingly managed to get their way in a bunch of countries.
The upshot is that, like the US and UK, RCT methodology and the status quo (SQ) have a “special relationship.” If you take RCTs seriously, you have to label virtually any radical departure from the SQ as “unscientific.” After all, if the change is radical, it won’t be feasible to run an RCT. And if RCT is the only scientifically respectable methodology, then every radical departure from the SQ is scientifically baseless.
Ironic, given all the publicity about how wonderfully “radical” the RCTers are.
Still, there are multiple ways to interpret the special relationship between RCT and SQ. Let’s start with the mildest, then intensify, step by step.
1. Big changes from the SQ can’t be justified using RCTs, but neither are they unjustified. As far as social science is concerned, it’s a “Whereof one cannot speak, thereof one must be silent” situation. In other words, RCTs tell us to be agnostic about big deviations from the SQ.
2. Big changes from the SQ can’t be justified using RCTs, so we should, by default, be skeptical. That’s how the FDA would react to new untested pharmaceuticals, right?
3. Big changes from the SQ can’t be justified using RCTs, so we should expect them to fail – and regard their proponents as charlatans. Even if, by some miracle, they happen to be right, their methodology is reckless.
4. Big changes from the SQ can’t be justified using RCTs, so we should expect them to be disastrous – and regard their proponents as monsters. You would practically have to be a psychopath to blindly push for big social changes.
If you are a staunch RCT person, however, positions 2, 3, and 4 all suffer from a common problem: None of them has ever been justified by an RCT! There has never been an RCT showing that big changes sans RCTs merit skepticism. There has never been an RCT showing that big changes sans RCTs typically fail. And there has never been an RCT showing big changes sans RCTs typically end in disaster. Indeed, as far as I know, there aren’t even any old-fashioned observational studies supporting these conclusions.
Should we then retreat to position 1? It too suffers from a dire problem. Namely: If you don’t believe that changes supported by RCTs are, on average, better than changes not supported by RCTs, why do you support RCTs in the first place?
In short, there’s a dilemma of methodological advocacy: You can be enthusiastic, or you can apply the methodology. But not both. The randomista crusade is either hypocritical or stillborn.
Is there any way to escape from this dilemma? Yes, but only with repentant methodological humility. Admit that the real foundation of science is just common sense. And common sense tells us that RCTs are the most helpful way to advance our understanding of extremely narrow questions. But when you ponder bigger questions, RCTs are just one intellectual input out of many. Including the bigger question of, “When should we dismiss people who fail to use RCTs as charlatans?” For pharmaceuticals, the right answer is “often.” For economic growth, in contrast, the charlatans are those who dismiss everything we’ve learned without RCTs.
READER COMMENTS
Scott Sumner
Jan 13 2022 at 12:09pm
Richard Rorty would wholeheartedly agree with this post.
Dylan
Jan 13 2022 at 1:02pm
Perhaps I am missing some subtlety here, but #2 does seem pretty supported by the data we have for pharmaceuticals. 1) We know that fewer than 1 in 1000 drug candidates will be beneficial. 2) We know that theory doesn’t help us with problem number 1, plenty of drugs that seem like they should work, don’t. 3) We know that many of the drugs that were used before RCTs were common, don’t hold up to scrutiny of an RCT. This leads me to believe I should be highly skeptical of any drugs that haven’t gone through an RCT (and that’s before we get to the bias that most experimental drugs that expect to make it to market will go through RCTs, so if someone is trying to sell me something today that hasn’t I’m going to be even more skeptical).
None of this necessarily translates into other domains. I think our knowledge of physics is a lot more developed than biology, and would be much more confident of a prediction made in that field without the benefit of an RCT.
Oskar Mathiasen
Jan 13 2022 at 1:16pm
If you don’t believe that changes supported by RCTs are, on average, better than changes not supported by RCTs, why do you support RCTs in the first place?It seems pretty natural to have the preference ordering “thing the RCT showed to be non effective” is (in expectation) less effective than “thing no RCT is done on” is (in expectation) less effective than “thing the RCT showed to be effective”.Resulting in the RCT conserving expected effectiveness.Then running a RCT has value according to the value of the information it provided.
TGGP
Jan 16 2022 at 10:42am
Yes, that quote from Caplan made no sense to me either.
Mark Z
Jan 13 2022 at 3:18pm
What is it that RCTs are supposed to have changed? Is it A) they’ve allowed us to confidently test certain hypotheses, while leaving others still in the dark? or B) they’ve allowed us to discover that in areas where applicable, RCTs almost always support the null hypothesis and thus should lower our prior on any change having an effect? Interpretations 2-4 seem to follow from B, while 1 seems to follow from A. Perhaps I’m missing something, but there seems to me to be no reason why RCTs should change how we view other kinds of evidence in settings where RCTs aren’t possible, unless the results of RCTs have a systematic relationship to the results of non-RCT experiments (e.g., if, when both an RCT and a different method are used, RCTs consistently find lower effect sizes, then that convince us other experiments have a bias). Otherwise, aversion to non-RCTs doesn’t seem to be so much an epistemological phenomenon as it is researchers looking for their keys under the street light (that is, focusing on questions we can actually get good answers to; but that doesn’t affect the relative weight we should give to the competing hypotheses for questions we can’t answer as well).
Dylan
Jan 14 2022 at 8:39am
Well said.
Mark Z
Jan 14 2022 at 12:18pm
Thanks.
Matthias
Jan 14 2022 at 6:36am
Capitalism does well even when introduced gradually and piecemeal. More capitalism often helps at the margins.
(Not always, since some regulations are there to fix up other regulations, and removing the former without removing the latter might not always work.)
So RCTs’ bias in favour of gradual evolution from the status quo might not be a bad thing.
Similarly, to give RCTs a chance to work in the area of policy at all, we need to push policy decisions to as low a level as possible.
If a certain policy is decided at the municipal or county level, you can imagine running an RCT. Even lower levels are even better. Policies decided at the national or supranational level are pretty much impossible to do an RCT with.
That observation is related to the principal of subsidiarity. Hopefully, a familiar friend to libertarians.
Phil H
Jan 14 2022 at 10:50am
One way to think about it would be to divide economics into a prescriptive and a descriptive science. RCTs would be the beginnings of a rigorous empirical foundation for the descriptive science, which will hopefully one day inform the prescriptive side much more fully. But for now it’s pretty far from being able to do that.
Thomas Lee Hutcheson
Jan 15 2022 at 9:20pm
“In other words, RCTs tell us to be agnostic about big deviations from the SQ.”
Conservatives have been saying this for millennia. And they are right.
Thomas Lee Hutcheson
Jan 16 2022 at 7:01am
The problem with RTC at least in the FDA incarnation is that the are not embedded into a broad enough cost-benefit framework. The do not a) take account of the time value of the process b) they do not take advantage of external costs and benefits, c) they seem (I’m not close enough to really know) to over-weigh side-effects costs, d) the do not give adequate weight to evidence from other countries.
Dylan
Jan 16 2022 at 11:13am
I’m not sure what level you’d consider over-weighting, but the vast majority of drugs fail due to lack of efficacy. In Ph III trials (which is where most safety signals show up, due to their larger size) fewer than 1 in 5 trials fails because of safety. More than half fail due to lack of efficacy.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6092479/
Carolyn
Jan 18 2022 at 9:34pm
I agree with Hutcheson. It is not like all clinical studies have equal access to the ability to conduct RCTs. Who and why a clinical study is monetarily supported for a randomized clinical trial can depend greatly on who will profit from a positive outcome trial result. Look at those rare diseases that few people have who could greatly benefit from pharmaceutical studies that could lead to and provide effective treatment. The potential for small company profit margins due to limited demand discourage financial support of RCTs. Compare this situation to the race to compete with clinical trials for the potential covid vaccines where there was a huge potential for profitability. Those large businesses with funds readily available were able to participate in the race. The other businesses who might have had a better vaccine idea were not able to participate. Have we allowed the probability of profit to limit our ability to discover new ways to meet people’s needs. If an RCT is required for the use of studied theory are we not limiting our choice of what theories may provide a better society or economy in the future?
Comments are closed.