The Dirty Laundry of Instrumental Variables
By Bryan Caplan
Are instrumental variables (IV) estimates really superior to ordinary least squares (OLS)? Most high-status empirical economists seem to think so. Meta-analyses often treat IV as presumptively superior to OLS. Yet when you ponder IV output, it’s often simply bizarre.
Rose and Betts, “The Effect of High School Courses on Earnings” (Review of Economics and Statistics 2004) is a case in point. The paper’s very good overall. But take a look at their IV estimates for the effect of coursework on earnings. (Columns 2 and 4 adjust for student ability; columns 1 and 3 don’t).
Readers eager to find an effect of curriculum will eagerly point out that the IV estimate implies that algebra/geometry raises adult earnings by 9-10%. But the IV results also imply that advanced algebra reduces adult earnings by 9-10%. (This result is only statistically significant after controlling for ability, but still).
If you look only at point estimates, and ignore statistical significance, the results are even harder to believe. You get big positive effects of secondary physics and third- and fourth-year foreign languages. But you also get a massive negative effect of calculus. Calculus!
You could accuse me of cherry-picking (lemon-picking?) some exceptionally odd IV results. But in my experience, Rose and Betts is typical. And if published IV results are implausible, the unpublished results are probably far worse.
A staunch empiricist could admittedly object that we shouldn’t call statistical output “bad” merely because it contradicts our theories. I’d reply, though, that “IV estimates are better than OLS estimates” is itself a theory. The main way to test this theory is to race IV versus OLS on questions where we are already confident that we know what the right answer is (or at least what the wrong answer is). By that standard, the privileged status of IV seems unjustified.
Please share your dirty laundry about instrumental variables in the comments. To avoid confirmation bias, please also share your clean laundry. 🙂