I noticed continued resistance to survey evidence in the comments. Frankly, it’s hard to believe there’s anyone left in social science who doesn’t take surveys seriously. Friedman’s original arguments against survey evidence were half-baked, and no one’s come up with any better objections since.
Yes, there are problems with surveys, but there are problems with all empirical evidence. And most of the data that economists use “instead of” surveys are themselves collected using… surveys! Where do you think unemployment numbers come from – hiring private eyes to follow people around all day? For goodness sakes, even if we did hire P.I.s, the data would reflect what the P.I.s said people did. It would be a second-hand survey, but a survey nonetheless.
READER COMMENTS
rapscalion
Mar 7 2010 at 6:14pm
It depends what the survey data is for. If you are studying opinion per se–e.g. Presidential polling–then it’s pretty good. Your studying opinions by simply asking for opinions.
For income and other basic econ data, I think it’s overrated. Comparing IRS data to BLS data on income, for instance, shows big differences. Most people are too stupid and/or ignorant to give accurate answers.
I’d say it’s the most unreliable about questions of motivation: asking WHY people do what they do.
People don’t know themselves that well.
John Thacker
Mar 7 2010 at 7:40pm
But some sort of surveys are clearly better than others.
No-doc home loans and home loans where someone produces income documentation are both sorts of surveys. Bryan might say the latter is simply surveying what a company paycheck says about someone’s income, so it’s only a second-hand survey. And yet one is considerably more accurate than the other.
Yes, but the motivation for lying is somewhat different in the two cases.
Simply because both are surveys and because it’s impossible to avoid all bias doesn’t mean that some are worse than others.
Loof
Mar 7 2010 at 8:14pm
Friedman’s half-baked and Bryan overcooks. For a good, simple overview of social research with pros and cons of not just doing surveys but different types of surveys see: Alan Aldridge & Ken Levine’s Surveying the Social World: Principles and Practice in Survey Research.
Norman
Mar 7 2010 at 10:27pm
I think the design of the survey is key. Whenever presenting survey evidence, I think the burden of proof always falls on the one citing the survey to show that it is incentive compatible. We could, of course, start from the assumption that people tell the truth unless they have a particular incentive to lie; but a more realistic assumption would probably be that people lie unless they have a particular incentive to tell the truth.
Nathan Smith
Mar 7 2010 at 10:37pm
Norman writes:
“A more realistic assumption would probably be that people lie unless they have a particular incentive to tell the truth.”
Too cynical. Most people have at least some moral qualms about lying and wouldn’t do it for no reason. People don’t have to answer surveys, anyway. If they do it, it’s typically because they like talking about themselves and/or they want to serve the cause of science. Either way it’s a reason to tell the truth.
A bigger problem is that questions are often poorly framed and data are often misinterpreted.
agnostic
Mar 7 2010 at 10:55pm
“but a more realistic assumption would probably be that people lie unless they have a particular incentive to tell the truth.”
Think through what you’re saying: for surveys that ask who a person is going to vote for — McCain, Obama, or Undecided — are you claiming that unless the survey is meticulously designed, those who say McCain really will vote for Obama, those who say Obama really will vote for McCain, and Undecideds are certain who they’ll vote for?
Indeed for a two-answer question, which many survey questions belong to, the “lie as default” strategy wouldn’t work — the recorder would just code your response as the other possible answer.
You must not talk much to people if you think that “lie as the default” is what most people in most situations have.
John Thacker
Mar 7 2010 at 11:04pm
I read Bryan’s post explaining why the ideas were “half-baked” as admitting nearly every criticism I’ve heard of surveys. Apparently Bryan also believes that “it’s hard to believe there’s anyone left in social science who does take surveys too seriously,” as well, considering how he views it as extremely obvious how and when to take survey results with a grain of salt.
And yet, far too often I read stories or papers where social scientists uncritically report survey data when there are enormous problems with self-interest in the results. Since social scientists so often trust survey data even in the cases where Bryan concedes that it’s flawed, skepticism is still entirely warranted.
Swimmy
Mar 7 2010 at 11:51pm
Even with the general skepticism toward surveys, I’m baffled that anyone is questioning their use in this particular example. Bryan uses survey data to show that people support social security, including young people. And every effort to reform social security (including in the favor of young people) has been resoundingly defeated, even when conservatives ran everything. With all the complaints, you’d think the data doesn’t completely and totally conform to our experiences of how people actually vote or something.
MernaMoose
Mar 8 2010 at 1:06am
it’s hard to believe there’s anyone left in social science who doesn’t take surveys seriously
And it’s hard to believe there’s anyone outside social science who takes the majority of them seriously, beyond anything more sophisticated than “who are you going to vote for?”.
I’ve seen way too many surveys that reek of bias, in order to support the “progressive” liberal left agenda. And they get reported by the MSM like they’re The Clay Tablets.
Europeans, for example, are “happier” than Americans. Remember that one, from not so long ago? The only “Americans” I know who believe that are liberals (the same people who tell me that, in spite of clearly different economic growth rates, people in Europe are just as rich and have just as many opportunities in life as Americans, on average — I don’t believe it). The outcomes of so many surveys depend almost entirely, on both what is asked and how it’s framed.
Very, very few surveys are designed by libertarians. Which is a key reason this libertarian tends to doubt the validity of many surveys.
MernaMoose
Mar 8 2010 at 1:12am
What do you suppose surveys of ObamaCare would look like, if the questions were generated by libertarians?
And is it possible, even remotely, that the questions libertarians would ask (and nobody else will), might have some impact on public opinion?
One beef I have with surveys is that what questions get posed, can impact public opinion all by itself.
The truism “correlation is not causation” aside, the fact that we have conducted a survey is not evidence that we have asked the right questions.
Ocean
Mar 8 2010 at 9:09am
Having worked in Market Research, I’ve seen how powerful surveys can be if carried out correctly, but also how misleading they can be if not (usually due to faulty survey design). Many times I find myself asking what questions (or answers) are left out just as much as what was included.
I find it hard to doubt the accuracy of the social security survey very much. I didn’t see the questions themselves, but even if it were off by several percentage points it wouldn’t change the outcome, and the arguments against its use were all abstract.
Lately I’ve become more and more enamored with prediction markets. Ask people to put their money where their mouth is, and you start seeing much more honesty.
PrometheeFeu
Mar 8 2010 at 7:42pm
I think surveys are great under very specific circumstances: If you ask people simple clear factual questions within a setting in which they have a low stake in the question inciting truthful answer.
Are you unemployed? Bad! Most people don’t understand the definition and they don’t like to think of themselves as unemployed.
Are you looking for work? A little better. But then again, you have the issue that people might not want to self report as not looking for work if they don’t have a job.
Anonymous survey: Have you sent a resume to at least on company in the past 3 weeks? Good! People can understand that, their anonymity lowers the stakes and removes the incentive to lie.
In my opinion, most topics are amenable to surveys only in very non-trivial ways. And I’m not sure I trust a survey I have not read myself to not be poorly designed.
Comments are closed.