Sampling Bias on a Plane
By Bryan Caplan
The numbers are in: SoaP’s domestic gross for opening weekend was $15M – less than a bunch of movies this year that I’ve barely heard of.
How is this possible? The answer, of course, is sampling bias. Your friends are not a random sample of Americans. In fact, they could easily be a highly unusual sample of Americans (or maybe not Americans at all). Well, to be honest, if you’re reading this blog, it’s virtually a sure thing that your friends are a highly unusual sample.
When you generalize from a random sample to a population, your inferences are highly likely to be correct. If 100% of people randomly sampled know who the current U.S. president is, you can reasonably bet that almost everyone knows who the current U.S. president is. But if 100% of the people you know say they plan to see Snakes on a Plane, you need to do some hard thinking. How close to a random sample are my friends? How does their education/age/gender/IQ/geekiness/etc. compare to the average American’s?
If you realize that your friends are far from average in a bunch of ways, you’ve then got to ponder whether these abnormal characteristics predict abnormal behavior. And needless to say, attending Comic-Con and GenCon is abnormal behavior.
So it’s no surprise that SoaP fell so far short of expectations. But I have to confess that, though I heavily adjusted for sampling bias when I guessed its opening gross, my adjustment was way too small. I reasoned: “If you believe the geeks, it will make $100M this weekend. That’s way too high – $50 M. OK, make that $45M with the R-rating.” Perhaps before I guessed I should have taken a second look at my pictures from GenCon!
My favorite example of sampling bias used to be people who believed that the 1972 Nixon election was fixed because “I don’t know anyone who voted for him.” Thanks to Snakes on a Plane, I can now update my lectures.
P.S. I will still be seeing SoaP later this week. Now is no time to distance myself from Nerd Pride.