Here’s The Economist:
In recent years medicine, psychology and genetics have all been put under the microscope and found wanting. One analysis of 100 psychology papers, published last year, for instance, was able to replicate only 36% of their findings. And a study conducted in 2012 by Amgen, an American pharmaceutical company, could replicate only 11% of the 53 papers it reviewed.
Now it is the turn of economics. . . .
In a paper just published in Science, Colin Camerer of the California Institute of Technology and a group of colleagues from universities around the world decided to check. They repeated 18 laboratory experiments in economics whose results had been published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014.
For 11 of the 18 papers (ie, 61% of them) Dr Camerer and his colleagues found a broadly similar effect to whatever the original authors had reported. That is below the 92% replication rate they would have expected had all the original studies been as statistically robust as the authors claimed–but by the standards of medicine, psychology and genetics it is still impressive.
Any thoughts on why economics is more scientific than science?
PS. I’m being provocative here. I actually think debates over what is or isn’t scientific are silly. The debates implicitly assume that science is a well understood concept, and we need only figure out whether economics fits into the category, whereas the opposite is more nearly true. There is also an implicit assumption that science is better than non-science, which is like saying the planet Venus is “better” than the planet Mars. By what criterion? Why should economists care if it’s a science?
North and South Korea both have enough “science” to build a nuclear bomb (although the South chose not to do so) but the two governments clearly have different levels of economic expertise.
READER COMMENTS
Daniel R. Grayson
Mar 20 2016 at 5:43pm
There’s a lot of economics not accessible to experiment, so one also needs
a criterion for judging whether that part of economics is “scientific”. As a first
approximation, one may say that economics is scientific if two rational economists in
honest disagreement can resolve their dispute, at least in principle. This would be
done by examining the details of the arguments on both sides, using criteria for
correctness that are jointly held.
The reason to care is that one wants to be able to trust the results and use them to
help society. Can economists divided into camps affect legislation? Should they be able to?
Reproducibility of experiments, the original topic, is related this way: if experiments in
the field tend not to be reproducible, one can predict future schisms as experiments
are repeated with contrary results.
Jack PQ
Mar 20 2016 at 5:48pm
Possibilities:
(1.) Economics papers are longer and more detailed than psychology or science papers, facilitating replication.
(2.) Experimental economists have strict no-deception rules and more generally, there might be less “noise” involved.
(3.) It is very difficult to publish experimental economics unless you are part of a well established and highly reputable lab group. In other fields, everybody does it.
(4.) Economists have a deeper respect for and understanding of the size and power of tests.
(5.) Economists are more likely to conduct robustness checks to rule out spurious significance.
I’m not saying these are absolutely true, but they seem plausible given what I’ve seen and experienced.
Andrew_FL
Mar 20 2016 at 5:49pm
Physics envy + Try-hards
Richard O. Hammer
Mar 20 2016 at 8:15pm
I wonder about the marketplace for publications in various disciplines. Given a particular subject, how many journals compete for submissions? Given this competition, how much can a journal demand careful (and thus reproducible) experimental design?
Might the Economist‘s reported difference in reproducibility be attributed to a greater number of journals operating in some disciplines?
Paul
Mar 21 2016 at 1:01am
Economics in a lab is only a small section of the current literature and certainly has only been part of the literature for a limit amount of time. Further, only areas such as game theory and behavioural economics are currently predominantly based on lab experiments. So I think it’s reasonable to expect experimental economics – though around for a while – is still testing the very core ideas and theories.
Now consider how much of the literature from last year in the fields of medicine, psychology and genetics is still testing ‘core’ or ‘fundamental’ ideas? I would say very few if any.
I suggest we are not comparing apples with apples. If we took only papers published if medicine, psychology and genetics which focus on widely accepted views/theories perhaps the replications would be significantly more robust. Perhaps more robust than in the economics literature.
Toby
Mar 21 2016 at 3:51am
I don’t think that there is any reason to suspect that economists are better scientists than most.
What I do suspect is that in fields such as psychology and genetics it is much harder to keep everything else constant because there is so much more to keep constant. This makes it much harder to replicate anything as the description of the experiments will be more likely to omit factors that are relevant.
And medicine, well, I suspect that the incentives are simply worse.
Unlearning
Mar 21 2016 at 6:54am
It’s not clear if TE is comparing like with like – are all of these psychology and medicine papers also experiments?
A recent paper found that most economic research isn’t replicable: http://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf
…but this includes statistical stuff, which is much trickier in econ.
Experiments in economics are usually pretty well focused on one or two mechanisms. The question is how generalisable the findings are, and if the environment has simply been designed to produce the require result (same applies to psychology).
Dw
Mar 21 2016 at 10:08am
A possible explanation: experimental economics is a newer field than medicine, genetics, and psychology. Economics still has low-hanging fruit in terms of lab experiments–as in experiments that confirm the obvious–whereas in the other fields, these types of experiments have been exhausted decades ago.
Eric
Mar 21 2016 at 10:46am
My candidate for working paper of the year is relevant here:
http://personal.lse.ac.uk/YoungA/ChannellingFisher.pdf
Basically, the asymptotics that are typically used to evaluate experiments hide some testable assumptions that are typically violated. These violations bias standard methods in favor of rejecting the null of no treatment effect. I would imagine that similar critiques would apply to other scientific disciplines.
David - Time Will Run Back
Mar 21 2016 at 11:43pm
Eyeglasses for the Brain
Human bodies have limits. Our ears have limits: We can’t hear ultrasonic sounds. Our eyes have limits: We can’t see ultraviolet light. We can’t see super tiny things. We can’t see super distant things. We invented microscopes to see small things better. We invented telescopes to see far things better.
Our brains have limits, too. People have lots of biases that fool them. The effect can be very strong. Being smart is not a sure protection against bias. In fact, the smarter you are, the better you are at coming up with creative excuses why your favorite bias is OK.
Confirmation bias is a major problem that leads us to be confident about things we should not be confident about.
Like microscopes and telescopes, we invented the scientific method to overcome limits in our brains, such as confirmation bias. The scientific method is a group of techniques that help us understand things that our unaided brains wouldn’t accept.
Some of the techniques are:
* Searching for a falsifiable hypothesis.
* Requiring greater amounts and more types of
evidence for claims that run contrary to
previously verified conclusions.
* Willingness to drop previous conclusions if the
evidence is strong enough.
* Double-blinding an experiment.
* Randomized controlled samples.
* Peer review
* Replication, preferably by suspicious
researchers with contrary biases.
Economics and psychology have a severe handicap in trying to use these techniques compares to physics: Randomized controlled samples are wildly unethical when dealing with human subjects. It’s OK to grab 100 rocks and try pouring hydrochloric acid on a randomly selected 50 of them. It’s NOT OK to grab 100 people and try imposing (say) Fascism on a randomly selected 50 of them. It’s NOT OK to grab 100 people and try sleep deprivation on a randomly selected 50 of them. This will always be a handicap for Economics and psychology…neither can ethically use the full set of scientific method techniques.
I like the scientific method but I like liberty a lot. Controlling the lives of unwilling human subjects in a totalitarian fashion is wildly anti-liberty.
I think the right attitude is to search for natural experiments and hope that the confounding variables are small enough to not ruin your conclusions. Or do contrived experiments with willing subjects in a lab setting that doesn’t much resemble the real life of humans and hope that we can learn something.
Either way, the soft sciences need to be permanently less confident about their conclusions. It’s the only correct response to the fact that economics and psychology can’t fully use the techniques of the scientific method.
— Extra paragraph for libertarian audiences —
That does not mean libertarians should be self-doubters while those leaning left and those leaning right should be confident. ALL should be self-doubters.
Why should economists care? Because they want to be right more often than wrong, and because they want to improve their answers as time goes on. The techniques I mentioned earlier help us make fewer mistakes. I don’t think it’s a matter of “science or not.” It’s a matter of the more of the techniques you employ, the more scientific you are. If you are nearsighted, reading is easier with glasses. If you have a human brain, overcoming biases is easier with the techniques I mentioned earlier.
Ram
Mar 21 2016 at 11:59pm
A simple model of replication:
Suppose researchers decide which studies to carry out by randomly sampling from a population of studies. Reduce each such study to a classical hypothesis test. The null hypothesis is true in a random study with probability n. Assume every study has power p and size s. A study is published iff the null hypothesis is rejected. A published study is replicated iff the null hypothesis is independently rejected in a follow-up study. The replication rate, r, is the probability of a study being replicated, given that it has been published:
r =
((s ^ 2) * n) + ((p ^ 2) * (1 – n))
————————————–
(s * n) + (p * (1 – n))
Note that r increases with p, and decreases with s and n. If the replication rate is higher in economics than in psychology, this suggests that economic studies enjoy some mix of (1) lower type I error, (2) lower type II error, and (3) lower base rate of true nulls.
Superior statistical practices ((1) and (2)) may play a role, though I doubt this explains much of the observed difference in replication rates (granting that this difference exists).
Plausibly, (3) explains the bulk of the difference between economics and psychology. This could be a good thing (economic theory guides researchers towards more a priori plausible alternative hypotheses relative to psychological theory), or it could be a bad thing (economists pursue more trivially true alternative hypotheses relative to psychologists).
I agree with your larger point, however. What matters is whether economics generates insight into, and understanding of, economic phenomena. Whether it is a science or not is a matter of definition, and being clear about the relevant definition would not help us to answer the question of interest. That economics has not produced knowledge characterized by the precision, accuracy or consistency of physics need not reflect badly upon economic method, but may simply reflect inherent challenges in generating knowledge of that kind in the economic realm.
MikeDC
Mar 22 2016 at 11:14am
I think the North vs. South Korea example reveals why science is preferable to non-science.
I don’t think the North Korean leaders are literally irrational, or even rational but ignorance of economics. Rather, they understand that while greater economic freedom would create greater prosperity for their nation, it could be harmful to their control. And ultimately their well-being.
Thus, economic science, applied in the big picture, is the application of truth for the benefit of mankind. Non-science is the rejection of these truths because they may damage individual (and/or in-group) wealth and status.
At the research level, this plays out differently. Economic lab research isn’t a very salable commodity at the moment (am I wrong about this?). Medical research is. One would think this results in stricter scrutiny, but it seems like an application of Gresham’s Law to me.
Lorenzo from Oz
Mar 23 2016 at 2:20am
“but the two governments clearly have different levels of economic expertise.”
The purposes of the governments are very different, which seriously dominates any question of economic expertise. It is, after all, not as if there is not a road map to mass prosperity away from a command economy — China displays it big time, even Vietnam, Cambodia and Laos somewhat.
The policy purpose of the North Korean economy is to extract a reliable surplus to a small elite, including the psychological benefits of status and dominance. It does that quite well.
Floccina
Mar 25 2016 at 10:05pm
Nutrition has to be the worst.
Comments are closed.