Social Science, Statistics, and the Garden of the Forking Paths
By Amy Willis
EconTalk host Russ Roberts has made no secret of his misgivings about high-level statistical analysis. So it’s no surprise that his skepticism is brought to bear in his interview this week with Columbia University’s Andrew Gelman. However, Roberts magnanimously starts the conversation by wondering aloud whether he’s gone too far in his skepticism. Maybe there are indeed things we can learn, and that we could not learn otherwise, via data analysis.
Gelman, a statistician, suggests that reliance on statistical significance is answering the wrong question…There is an extended discussion on the extent to which “p-hacking” is a problem in statistical research, as well as a fascinating thread on the prevalence of “priming.” (At the end of the conversation, Roberts refers to Brian Nosek’s replication project as “God’s work.”)
The real point of the conversation to me, though, are the big questions raised. Roberts, about half-way through, genuinely asks, “So, now what?” Are we to discard all data analyses and resort once again to pure theory? Can statistical analyses ever avoid becoming ideological cudgels employed to beat down one’s opponents? Should we reconsider the place of social science in policy altogether? What about what we consider to be social science? Is it enough to rely on your “gut” and be honest about it, as Roberts suggests?
These are just some of the questions I’m left thinking about after this week’s conversation. I’m not really comforted by Gelman’s contention that things would be better if only people had a better understanding of what statistical significance does (and does not) convey. I’m even less optimistic that more social scientists will go Gelman’s route and endeavor to better integrate theory into their data modeling. But I always aspire to be proven wrong…