Imagine a Question in social science.
The Question can be analyzed using an Obvious Method – a simple, standard approach that social scientists have used for decades.
The Question has a Welcome Answer – an answer that the typical social scientist wants to hear.
What determines the popularity of the Obvious Method?
Here’s my simple cynical theory:
1. If the Obvious Method yields the Welcome Answer, the Obvious Method will be popular.
2. If the Obvious Method fails to yield the Welcome Answer, the Obvious Method will be unpopular.
If #2 holds, then:
3. If a Non-Obvious Method yields the Welcome Answer, the Non-Obvious Method will be popular.
4. If no Non-Obvious Method yields the Welcome Answer, the Question will be unpopular.
My model clearly doesn’t explain everything. Some academics really do love methodological sophistication for its own sake. But I still think my model explains a lot of question-by-question variation in researchers’ methodological finickiness.
Do you see what I see? Please offer disconfirming as well as confirming examples.
READER COMMENTS
Ross Levatter
Jan 24 2014 at 12:44am
Why does this remind me of a Darek Parfit thought experiment?
John Becker
Jan 24 2014 at 1:43am
#3: Macroeconomics of the John Keynes/Milton Friedman/Paul Samuelson variety.
TallDave
Jan 24 2014 at 2:22am
Another flavor of searching for the data you like in a noisy signal, a problem for scientific epistemology.
The way humans process information leads to epiphenomena like EVP, we’re very poorly equipped to perceive reality accurately without a lot of squinting and checking the rulebook. Evolution doesn’t select for intelligence, it heads blindly for local optima.
Garrett M. Petersen
Jan 24 2014 at 2:59am
You probably have the monopsony argument for the minimum wage in mind for #3, correct? That’s a clear example of an implausible theory getting more attention than it deserves because it yields a welcome answer.
I can’t think of too many unpopular questions as you’ve described them, maybe because economists are so good at coming up with non-obvious methods to find their desired results. “Are different economic outcomes for people in different racial groups related to their genetic characteristics?” is a candidate. I’m not sure how well it fits.
Kevin Erdmann
Jan 24 2014 at 4:06am
We’re all probably susceptible to that.
Tyler recently posted a chart of mine that showed a kink down in the trend of teen employment in 6 of the 7 series minimum wage hikes. It was clearly not the most sophisticated analysis ever performed.
But, I got a chuckle from commenters and economists whose response was (paraphrased):
“We shouldn’t even be looking at this. The model here is so underspecified that he can’t begin to make inferences….and anyway, I can tell just by eyeballing it that the drops in employment are all because the minimum wage hikes coincide with recessions.”
Richard O. Hammer
Jan 24 2014 at 7:40am
This post seems to me to make a valid, good point. But must we feel cynicism in response? Instead, can we say, “Yes, that is right”, and proceed with our knowledge of social science thus enhanced?
People have goals, that is biases. Recall the advice: If at first you don’t succeed, try, try again! Indeed we try again and again to find ways to reach our goals. And that is good.
What may be bad, I propose, is the notion that scholars do not have motives, goals, or biases. I believe the work on preference falsification by Timur Kuran may be relevant.
Daniel Kuehn
Jan 24 2014 at 8:25am
Kevin – not sure if you’re thinking of me, but I said something like that. Why did you get a chuckle out of that?
It seems like an important problem.
If you can’t justify why you know the drop is due to the minimum wage vs. an obvious alternative cause, why are you making inferences from the data? You shouldn’t chuckle at someone that expects that. That is a very minimal expectation. In fact that’s Bryan’s “obvious method”. We don’t even get into particularly fancy stuff with that complaint.
Daniel Kuehn
Jan 24 2014 at 8:29am
I think this is what a lot of people assume about a lot of other people but don’t think is true of themselves. Which opens one of two plausible possibilities (well and I suppose any combination of the two): we’re all cynical of each other but Bryan is wrong, or we’re all self-deluded but perceptive of others and Bryan is right.
I tend to think Bryan is wrong. The critiques of the “obvious methods” are very real, after all. It’s not like technicalities in method have built up randomly. They address very easily verifiable problems in earlier methods. And my experience is people care about those problems irrespective of the answer.
I do think solutions often introduce their own problems (as we’ve seen with Bryan’s posting lately on IVs and the literature on weak IVs), which some people are less aware of than the problem they were trying to solve in the first place. That can introduce error into analysis. But I don’t think it has anything to do with getting a popular or unpopular answer. One of the nice things about having a group of people obsessed with neat methodological tricks is that to the extent that’s what they care about they are less invested in the result itself.
Daniel Kuehn
Jan 24 2014 at 8:31am
On an autobiographical note, I am pretty much on my way to rejecting my prior (my preferred answer) on my first dissertation chapter because I trust my method and it’s not giving me the answer I expected or wanted.
I also rejected the “popular” answer – the one that I wanted to see – many years ago when I first got introduced to the empirical literature on the minimum wage, because I thought the methods addressed real problems in other studies.
Maybe I’m deluding myself elsewhere of course, but these are two cases where I embraced unpopular answers from non-obvious methods because I thought those non-obvious methods addressed very real problems in the analysis.
April Harding
Jan 24 2014 at 8:48am
I believe a variation of this preference foundation underlies much of the passion for RCTs in development economics. The RCT methodology supports framing development questions as ones with simplistic “intervention” answers.
– deworming children via schools is a development “best buy”
– free distribution of bednets via mass campaigns can control malaria
– removing fees for services can improve population health in poor countries.
The preference foundation is slightly different to the one you propose; researchers’ preferences for RCTs are driven not so much by the answer they want, as by the kind of answer (shedding light on the effectiveness of a simple unvarying recipe which aid agencies and charities can fund).
dave smith
Jan 24 2014 at 9:45am
Daniel, what you should want and expect from a dissertation is for it to be signed. Don’t waist your life in grad school looking for answers you want. That is what tenure is for.
In other words, get your thesis done.
Daniel Kuehn
Jan 24 2014 at 9:49am
dave smith –
I’ll be out fairly quickly. Sometimes I worry I’m getting out too quickly, but then I remember I have a baby and a mortgage and don’t worry about that any more.
And I thought the whole moral of my story was that I am NOT looking for the answers I want! That I’m quite happy getting an answer that I didn’t want because I trust it.
Kevin Erdmann
Jan 24 2014 at 9:56am
No, Daniel. Who’s on first, what’s on second, and I Don’t Know’s on third…….
Daniel & I will be here all week, folks. Tip your waiters.
David R. Henderson
Jan 24 2014 at 10:08am
@Daniel Kuehn,
Good, thoughtful points about methodology.
You’ve got me curious: what was your prior in Chapter 1 and what are you finding? Of course, that’s if you’re willing to say.
John Soriano
Jan 24 2014 at 10:53am
I don’t know him and I haven’t read the majority of his research so I don’t want to unequivocally question his motives, but it seems like recently any time an econometric paper that shows negative effects of minimum wage increases comes out, Arindrajit Dube will write a response that changes up the methodology and gets his “Welcome Answer.”
For instance, on the recent Meer and West paper, he performed a falsification test that only holds if you exclude business cycle controls that Dube himself has argued are essential.
Daniel Kuehn
Jan 24 2014 at 11:00am
David –
The paper is on the employment and earnings effects of job creation tax credits (and actually investment credits… I’ve recently found out they were phased in using the same selection rule so I can’t distinguish the two, which is fine I guess).
My prior was that they would create jobs and raise wages. I have a good identification strategy – an RDD model. But one thing lacking in the existing literature on it is a way of dealing with displacement effects (in other words, person A gets the job from the tax credit by displacing person B who was not eligible for the credit). I can deal with that (at least within-county displacement). I expected that would reduce the effect somewhat of course, but I was sure even after accounting for displacement the credits would still generate jobs.
So far, they seem to reduce employment. Displacement appears to be a big problem.
There is one other explanation I’m investigating now. You have to create full time jobs to get the credit, so it is possible that I’m seeing a negative employment effect because part time jobs are being replaced with full time jobs. I’m investigating that now with individual level data. So in the end, it may create full time jobs and destroy more part time jobs, in which case it would be interesting to look at the impact on total hours.
I’m not sure how it will all shake out in the end, but I am definitely less confident in policy than I was before I started this.
Daniel Kuehn
Jan 24 2014 at 11:11am
John Soriano –
I think the Meer and West response to Dube is a little confused. The big difference between the two papers is the use of contiguous county pairs. He can’t use the business cycle control (total employment) because that’s the dependent variable for Meer and West and he’s reproducing their results.
I’m not sure where he argues they’re essential, although he does note the problem of not being able to include them in Meer and West. The big concern he has is the spatial heterogeneity, and that’s why he reruns their analysis with the county pairs. When you do that, the results drop out.
Daniel Kuehn
Jan 24 2014 at 11:19am
John Soriano –
And the contiguous pair matching itself really is a business cycle control. That’s part of the appeal of doing it. If you have two contiguous counties then differencing out the change in the comparison county should help to difference out the business cycle effects that are hitting that area (along with anything else going on that’s common to neighboring counties). That’s part of the whole appeal of rejecting the Meer and West fixed effects approach and doing a DID approach instead.
Meer and West – in their original paper and in this appendix – seem to think that what they’re doing is a DID, but it’s really not unless you’ve got a matched comparison group like Dube does. Meer and West are implicitly assuming that any state can be a comparison group to any other state. That’s not a DID, that’s just regular old fixed effects model.
MingoV
Jan 24 2014 at 5:52pm
I can think of far too many examples. I give one related to my field.
Many toxicologists are more idealists than scientists. They believe that man has invented and uses too many poisonous chemicals. They ‘prove’ this by giving lab rats massive doses of a chemical. When toxicity is seen, they use a linear model to estimate the lowest dose that will cause toxicity. They scale that up to human size despite knowing that such scaling is inaccurate. This results in absurdly high ‘safe’ exposure limits such that the chemical cannot be used.
The linear toxicology model was disproved five decades ago. The correct model is that a chemical causes no toxicity until a certain concentration is reached. The toxicity curve from that point can be linear or exponential.
The animal scale-up model based by weight was shown to be wrong decades ago.
The more recent “non-obvious method” is to use genetically modified lab rats and mice that have defects that make them more susceptible to certain chemical classes. By selecting the ‘right’ modification, biased toxicologists can ‘prove’ that almost any chemical is toxic. These toxicologists thrive in California, the state with more banned chemicals than any other.
Michael Tontchev
Jan 24 2014 at 6:56pm
Daniel – I, too, wonder why he got a chuckle out of the critiques of his implications about the effects of minimum wages. Unfortunately, he was joined in by a choir of Austrian economists who supported this poor empiricism because it confirmed their policy prescriptions (and I say this as a person who is strongly influenced by the Austrian school). Imo if a given tool is bad, it doesn’t become good when it supports your side.
Kevin Erdmann
Jan 24 2014 at 9:28pm
Daniel & Michael,
The first sentence of the reaction I paraphrased is perfectly reasonable. The second sentence is an ironic refutation of the spirit of the first sentence.
Somebody got it. Right?
Dan
Jan 25 2014 at 12:08am
Maybe it’s because I read too much Steve Sailer, but the first thing that springs to mind for category 4 is the whole cluster of issues surrounding intelligence (its measurement, value, heritability, etc, let alone the fraught question of racial/gender differences).
You don’t see many people trying to make their career by investigating these sorts of questions too rigorously, that’s for sure.
Steve Sailer
Jan 25 2014 at 5:37am
Immigration’s effect on standards of living, obviously.
Theory (the Law of Supply and Demand), practice (e.g., the economic history of California), and bias (e.g., the lobbying of Mark Zuckerberg and the big growers for more immigration) all suggest that immigration lowers wages. But, a huge fraction of economists cling to a handful of studies (e.g., Card’s of Miami after 1980 and Peri’s of California’s before 2007) that attempt to muddy the obvious answer.
Obviously, ceteris wasn’t paribus in Miami from 1980-1984 due to the Scarface/Miami Vice influx of cocaine into the economy, and in California before 2008 due to subprime mortgages. But economists aren’t very interested in running reality checks when they’d rather affirm the empirical excellence of immigration.
Daniel Kuehn
Jan 25 2014 at 11:14am
Kevin Erdmann –
Right, but I’m going off of what I actually said and not your paraphrase. What problem do you have with what I actually said.
The irony of your second sentence turns on “are all because of”. If “are all” was actually “are partially”, then there’s nothing to chuckle about and you have some justification to do.
Now did anyone actually say your second sentence or are you just making things up to avoid the issue? I certainly don’t think I said the second sentence, so I just ignored it and restated what my concerns are with it.
luis mejia
Jan 25 2014 at 2:23pm
Steve Sailer sees immigration as a social process that obviously lowers wages.
He, and many like him, are entitled to their model driven opinion, of course.
However, I would like to challenge the usefulness of a model that isolates a process like migration from other relevant processes such as the effect of U.S. policies and economic activities in other societies. Studying migration in isolation of relevant macro political and economic variables obviously gives you the answer you want, although that answer will work only in a mind-made world kept in ideal isolation.
On the other hand, I’d like to remind Mr. Sailer that in the 1970s and 1980s the city of New York was a huge slum with pockets of prosperity here and there. It was us, immigrants, who rebuilt and repopulated the city making it economically viable for others to move in. Immigrants didn’t force salaries down in Wall Street; on the contrary, immigrants created the environment for those salaries to go high, real high.
He mentions Miami as an example of bad migration. I beg to differ. Miami was a sleepy, backwater kind of place until the Cubans first and other Latinos later made it the vibrant economic hub it is today. That some of those immigrants were probably drug traffickers or, worse, corrupt officers of governments allied to the US who found in Miami a heaven to enjoy their stolen money cannot be denied; it doesn’t mean most Latinos in Miami are dishonest people.
Was it M Friedman who said that a model didn’t need truthful assumptions if it produced the right predictions? In reality, then, we are not talking of models per se, we are talking about assumptions, and given the appropriate assumptions we can prove and disprove that immigration is good or bad for the economy without getting any closer to the real world.
Steve Sailer
Jan 25 2014 at 5:50pm
“He mentions Miami as an example of bad migration.”
You are missing the point. David Card famously cited the fact that after the positive labor supply shock in May 1980, wages in Miami did not go down over the period to 1984 relative to several other cities. This is widely cited by economists as proving that the Law of Supply and Demand does not apply to immigration.
I’ve pointed out that ceteris was famously not paribus in Miami: Miami in 1980-84 was experiencing a world famous (e.g., Scarface and Miami Vice) economic boom driven by the rise of Miami’s cocaine industry in the early 1980s. Here’s the 1981 Time mag cover story “Paradise Lost.”
http://randompixels.blogspot.com/2011/07/way-we-were-miami-in-1980smariel-murder.html
Kevin Erdmann
Jan 25 2014 at 6:34pm
Daniel,
I never said you said it. You said you said it.
After seeing those types of comments, I went back and did some more work to try to see how much of an effect recessions had on employment levels after minimum wage hikes, and I posted some of that on my blog.
Kevin Erdmann
Jan 25 2014 at 6:43pm
Being a financial speculator is helpful in fighting this tendency. Profits are generally only available on issues where other people are wrong and obstinate about it. Markets tend to be very efficient, so people really have to have universally thick blinders on about something for there to be dependable mispricings.
I need people to be obstinate. I don’t really want them to change their minds. I post online in order to reflect my thoughts off of others, to decide if they are firm about things for reasons that seem reasonable to me, and to change my own mind before I blow my money on a bad position.
It has been a revelation for me. I’m not saying I’m any better at it than anyone else. I probably started out being really stubborn. But, it has moved me forward from where I started.
Simon Cranshaw
Jan 26 2014 at 4:12am
Steven Sailer, surely that claim is only possible from a natitivist point of view. If all participants are considered I think standard of living can be expected to rise as obvious theory would predict.
Simon Cranshaw
Jan 26 2014 at 4:13am
Steven Sailer, surely that claim is only possible from a natitivist point of view. If all participants are considered I think standard of living can be expected to rise as obvious theory would predict.
Christopher Chang
Jan 28 2014 at 8:53am
@Simon: consider that the Chinese Communist Party has sovereignty over both mainland China and Hong Kong, and to the extent it’s exposed to public opinion, it’s mainland opinion that matters. Yet it continues enforcing immigration restrictions between the two areas, and seems to be doing a reasonable job of letting Hong Kong export its prosperity under this framework. It seems to think that keeping a relatively undisturbed working copy of Hong Kong around is “globally”, not just locally, utility-maximizing.
Michael Crone
Feb 1 2014 at 10:08pm
This story http://en.m.wikipedia.org/wiki/David_M._Fergusson
on the link between abortion and mental health either confirms or opposes Bryan’s thesis. It’s confirmatory because the study’s unpopular results made it more difficult to publish, but opposes the thesis in that the researcher worked to publish findings that he himself did not desire to find.
My personal experience is that ecologists show a particular lack of interest in studying how increasing CO2 levels help crop growth and a noticeable bias to showing harms rather than benefits of increased CO2 (or any environmental change for that matter).
[broken html fixed. I suggest using the button for links above the Comment box.–Econlib Ed.]
Comments are closed.