What’s the correlation between results from observational studies and results from randomized controlled trials? Medicine’s the obvious place to start, but if you know of any research on any topic that addresses this question, please share.
May 11 2016
Ralph Hawtrey has always been in the shadow of Keynes, but might well have been the superior macroeconomist. I think you could also argue that Hawtrey's model of macroeconomics was more "Keynesian" than the General Theory, in the sense of being closer to what economists of the 1990s and early 2000s meant by the term "K...
May 11 2016
A book review by Samuel Goldman on The American Conservative alerted me to "We Can Be Heroes: The Radical Individualism of David Bowie" by Robert Dean Lurie, a short e-book whose title is self-explanatory. I shall confess I wasn't much of a Bowie fan before he passed away a few months ago. But when I listened to his...
May 11 2016
What's the correlation between results from observational studies and results from randomized controlled trials? Medicine's the obvious place to start, but if you know of any research on any topic that addresses this question, please share.
READER COMMENTS
Luke Muehlhauser
May 11 2016 at 12:49am
Chapter 3 of Deeks et al. (2003) and the systematic reviews included there:
http://researchonline.lshtm.ac.uk/8742/
One of the included reviews was updated in:
http://onlinelibrary.wiley.com/doi/10.1002/14651858.MR000012.pub3/abstract
Also:
http://onlinelibrary.wiley.com/doi/10.1002/14651858.MR000034.pub2/full
And:
http://eprints.ioe.ac.uk/3685/
LemmusLemmus
May 11 2016 at 1:06am
An older entry from social psychology:
http://www.sakkyndig.com/psykologi/artvit/anderson1999.pdf
Richard O. Hammer
May 11 2016 at 5:41am
If I understand your question, I venture a correlation: both (observational and RCT) are seeking.
A difference, on the other hand, is that RCT is more advanced science — in the sense of already having a concept of what is not being sought. The difference is in the clarity with which the background of the question (the null hypothesis) has been defined.
Your question may be getting into philosophy of science. Are you already versed in that field? I could recommend a half dozen books.
Tiago
May 11 2016 at 8:33am
Great short post. This is the kind of question that once asked, we think: how can we not know this?
gwern
May 11 2016 at 12:07pm
Am I glad you asked. I have been compiling a bibliography of studies which explicitly or implicitly compare them: http://www.gwern.net/Correlation
The upshot so far is that there is no simple difference in average effect size, but the variance of comparisons is extremely high – the randomized effect vs the observational effect can easily be over or underestimated by a factor, or reverse sign.
So there’s no easy meta-analytic correction where you can just subtract d=0.5 and get a decent causal estimate, disappointing as this may sound; it’s more of a mixture model setup, where there’s a probability of ~1/3 that your observational estimate (no matter how precise or how many covariates you add) is totally worthless and entirely unrelated to the actual causal effect, and you will never know until you run a randomized experiment.
(This is maybe not as surprising as it sounds when it comes to economics and sociology, as they are totally confounded by genetics, but it’s still disappointing to see it so pervasive in medicine and other areas too.)
Evan
May 11 2016 at 1:38pm
There’s some focus on this issue in the realm of electric energy efficiency, where program administrators (e.g., utilities) are often required to demonstrate that their interventions have resulted in actual usage reduction. See, for example, discussion on page 19 of this report from the Department of Energy, where they compare the results of 17 programs under RCT and observational measurement methods:
https://www4.eere.energy.gov/seeaction/system/files/documents/emv_behaviorbased_eeprograms.pdf
I was unable to track down the original paper they cited (Allcott 2011).
Fabio Rojas
May 11 2016 at 1:46pm
On a recent Econtalk pod cast, one of Russ Robert’s guests (Adam Cifu, a physician) said that about 4 out of 5 correlational studies in a top medical journal stood up in RCT. Maybe you can hunt down the citation. Here is the link:http://www.econtalk.org/archives/2016/02/adam_cifu_on_en.html
Alex Tabarrok
May 11 2016 at 1:50pm
I cover this in a video at MRU
http://www.mruniversity.com/courses/development-economics/randomized-controlled-trials-part-two
Kevin
May 11 2016 at 3:46pm
Here’s one from Cochrane:
http://www.cochrane.org/MR000034/METHOD_comparing-effect-estimates-of-randomized-controlled-trials-and-observational-studies
gwern
May 12 2016 at 4:37pm
Lemmus: Anderson 1999 is about external vs internal validity, not randomized vs correlation. A correlation inside the lab and a correlation outside the lab can be considered a replication, but doesn’t tell you anything about what would happen if you randomized one of the variables. So I don’t think it’s relevant here.
Stuart Buck
May 13 2016 at 2:30pm
Thomas Cook has done a lot of interesting work on this issue recently, including within-study comparisons of the effects seen from the RCT comparison and from other methods:
http://intl-epa.sagepub.com/content/31/4/463.abstract
http://intl-epa.sagepub.com/content/31/4/463.abstract?patientinform-links=yes&legid=spepa;31/4/463
ZW
May 22 2016 at 4:11pm
Bloom, HS, Michalopoulos, C, and Hill, CJ. 2005. “Using experiments to assess nonexperimental comparison-group methods for measuring program effects.” Chapter 5 of ISBN: 978-0-87154-133-8.
Comments are closed.