The Econometrics Controversy
By Arnold Kling
Empirical microeconomics has experienced a credibility revolution, with a consequent increase in policy relevance and scientific impact.
The quote is from the paper, which is gated, and the link is to the abstract, which is not.
[UPDATE: ungated version here. ]
I think it is a really important controversy. I place myself on one side of it (the minority, naturally). Remaining comments below the fold.The question, broadly speaking, is whether sophisticated empirical work in economics is persuasive. Sophisticated empirical work consists of taking a single data set and using the best econometric techniques to arrive at estimates of the interesting parameters.
In contrast, when I have an empirical question, I look at a variety of data sources. For example, one interesting question is whether economic growth since 1800 has been much faster than economic growth for the preceding 1500 years. I believe that the answer is “yes,” based on a variety of indicators. Many of these can be found in chapter 2 of From Poverty to Prosperity, so I will not reproduce them here. The bottom line is that there are many ways to look at the question, and as far as I know, all of them point to essentially the same answer.
Another question might be, in mortgage performance is borrower’s equity an important determinant of default? I am convinced that the answer is “yes.” Again, this is not because of any one study, but because of a variety of studies that have looked at default rates relative to original loan-to-value ratios, relative to estimates of current loan-to-value ratios, studies that compared default rates under different economic conditions, etc.
Chris Sims represents the opposite school of thought. He believes in the triumph of state-of-the-art technique over weak data. I don’t know if it’s still true, but his professional reputation used to be very imposing. To question Sims was to make yourself look bad. I personally never saw the attraction. He may be gifted and clever, but I have never found him persuasive. If you’re one of those people who regards Sims as super-human, then you probably will not be on my side in the controversy.
Leamer is much more my champion in this. Here are two sample quotes from his paper.
Can we economists agree that it is extremely hard work to squeeze truths from our data sets and what we genuinely understand will remain uncomfortably limited? We need words in our methodological vocabulary to express the limits. We need sensitivity analyses to make those limits transparent. Those who think otherwise should be required to wear a scarlet-letter O around their necks, for “overconfidence.”
…Let’s face it. The evolving, innovating, self-organizing, self-healing human system we call the economy is not well described by a fictional “data-generating process.” The point of the sensitivity analyses that I have been advocating begins with the admission that the historical data are compatible with countless alternative data-generating models.
In an email about a paper in which I express my skepticism about macroeconometrics, Jeffrey Friedman asked me
Why are macro models so bogus? Is it because we just don’t have the right models yet, or because the world is inherently too complex to make sense of, or because there are too many factors at work at any given time, or because history never repeats itself (or some or all of the above)?
The problem is definitely not that we “just don’thave the right models yet.” I think that is close to Sims’ view–we did not have the right models in the 1970’s, but now we are getting there. I just cannot agree.
I think that “the world is inherently too complex” has some validity, but it is too much of a cop-out. We have to deal with the world as it is, as best we can.
I think the main issue is “too many factors at work at any given time.” In statistical theory, every time you add a new observation you get more information. That is because the theory assumes that the number of relevant factors is constant, and an increase in sample size gives you more variation in the relevant factors and thereby enables you to separate the influence of the different factors with greater precision.
In macro, adding observations does not help. When we get a quarter of more-or-less normal growth, the relevant factors do not vary by enough to provide any interesting news. There is not much noise, but not much signal, either. The statistically valuable observations are episodes like the Great Depression or the recent downturn. Unfortunately, in those cases, the list of potential causal factors is too long for the data to be able to distinguish. I think that the best count of potential causes of the financial crisis is well into the twenties. The list of theories of why the Great Depression was so deep and lasted so long is even more extensive. Thus, we are always in the position of having more theories than meaningful data points.
The main point of the Angrist-Pischke paper is that in microeconomics, the use of natural experiments has made econometrics more credible. As an empirical matter, I am not sure that this is true. See the appendix in Kling-Merrifield, where I found the “natural experiments” that supposedly prove a high return to education to be shockingly flimsy. If this is the sort of work that is supposed to take the “con” out of econometrics, it hasn’t.
I think if you step back and look at econometric history, you see the rise and fall of fads: simultaneous equation estimators, distributed lags, VAR’s, instrumental variables, regression discontinuity, and so forth. If you jump through the right hoops at the right time, you get your paper published. But what you publish is not reliable.
Most economists eventually see through the older fads. But while the technique is going through its boom phase, woe be it to the young economists who fails to jump on the bandwagon. It shouldn’t have to be that way.