Few economists bother to explain in writing why other people’s articles are a waste of time. Even fewer economists bother to explain in writing why entire journals are a waste of time. As far as I’m aware, in fact, there is only one outlet for economists to take journals to task: Dan Klein’s Econ Journal Watch. In the latest issue, Klein and co-author Pedro Romero dispassionately skewer the Journal of Economic Theory.
I suspect that JET fans will cry sour grapes, but Klein and Romero aren’t just whining. They have an original and thought-provoking empirical approach:
We examine the 66 regular articles appearing in the 2004 issues of JET and apply the three requirements of theory: Theory of what?, Why should we care?, and What merit in your explanation? We find that 27 articles fail the first test (Theory of what?) and 58 articles fail at least one of the three requirements. Thus, 88 percent of the articles do not qualify as theory.
Not convinced? Klein and Romero add:
Surely, the strongest defenders of model building would admit that the practice might go too far, into unjustifiable realms of creative writing. Economists will disagree about the “marginal conditions,” but every economist will affirm some marginal conditions, which imply the hazard of going too far. Every economist, therefore, should appreciate a watchdog effort, even if he objects to the watchdog’s test and results.
Yep.
READER COMMENTS
Gabriel M.
May 17 2007 at 12:13pm
Well, I wouldn’t throw around “thought-provoking empirical approach” so easily.
They act like they don’t understand the points made in many papers. They don’t deserve praises for that.
More methodological whining from crypto-Austrians. *Yawn*
conchis
May 17 2007 at 1:46pm
I’m no fan of the excessive mathematization of economics, and I wouldn’t go as far as Gabriel here, but certainly from a quick skim of the paper and the excel file detailing their results, it looks like they’ve been pretty harsh in the application of their tests.
It doesn’t seem that it would be particularly difficult to come up with applications for much of the work that they’ve essentially failed for not being applicable enough – in which case the criticism may largely amount to a complaint that people aren’t advertising their work well enough. That’s certainly a valid and important point, but it’s as much stylistic as substantive, and it’s a far cry from the implicit “all this stuff is useless rubbish”.
Barkley Rosser
May 17 2007 at 2:28pm
Following up on these two comments, let me note something else raised in comments over on Marginal Revolution, and which I as a journal editor am all too aware of.
Space constraints.
Journal editors generally encourage, indeed frequently demand, that authors pare down their papers. Journals face absolute space constraints, and authors tend generally to want to bloviate on and on about how wonderful their work is blah blah blah. While most editors usually like to have some discussion of “what is this about and why is it important,” some of that is also what can get cut in the pressure to meet space constraints. Editors and authors, especially in a place like JET, will often presume that readers of the papers know what the theory is about and what it applies to.
I have to agree with notsneaky over at MR and Gabriel M. here. In this particular article, the usually astute authors put some non-trivial egg on their faces.
Daniel Klein
May 17 2007 at 3:37pm
Saying that JET is unworthy casts doubt on the whole structure that has anointed it as worthy. If the article on JET is sound, or only roughly sound, it poses a challenge to what stands behind JET’s prestige.
Even if one has no particular fondness for JET, he may have fondness for other institutions whose anointment derives from the same structure as JET’s. Everyone craves establishment respectability, so criticizing the basic wisdom of establishment structures hits close to a lot of otherwise far-flung homes.
To go beyond broad bludgings and denunciations, we need serious exchange on the specifics at hand. Do those 66 JET articles satisfactorily answer Theory of what?, Why should we care?, and What merit in your explanation?
EJW is open to scholarly reply.
Gabriel M.
May 17 2007 at 3:56pm
Prof. Klein, why fail the authors of the paper whose summary is “We show the robustness of the Walrasian result obtained in models of bargaining in pairwise meetings.” on the “Theory of What?” point?
Should they have started with “This is a paper on prices.” to qualify?
conchis
May 17 2007 at 5:06pm
I think we’re potentially talking past each other a little here (though I don’t think that’s the only source of disagreement), when Dan asks whether “those 66 JET articles satisfactorily answer Theory of what?, Why should we care?, and What merit in your explanation?”
Much of the criticism of the paper is that this is the wrong (or at least not the most interesting) enquiry. The important issue as whether those questions are able to be answered, not whether they actually were. That’s the issue that speaks to the value of any given instance of model building – as opposed to whether authors have sufficiently advertised the value of their modelling to non-specialists.
conchis
May 17 2007 at 5:08pm
P.S. Dan, I’m not sure that insinuations of bias are helpful here. (Though nor is Gabriel’s name-calling.)
Pedro Romero
May 17 2007 at 5:15pm
Regarding space constraints in JET; a point raised by Barkley Rosser; there were articles such as: “Differentiating Ambiguity and Ambiguity Attitude” that has 16 (out of 40) pages for appendices. This points out how JET’s editors favor math proofs rather than persuasive explanantions about the explanandum. And this article is not the exception in JET.
Gabriel M.
May 17 2007 at 5:30pm
Yes. Sorry about the name-calling. I apologize.
That being said, everything else stands, I think. There’s little point in criticizing theoretical papers for being atheoretical because they lack applications or case studies or descriptions of real life phenomena.
And the categorization of papers in accordance with the three question was done with little, if any, charity.
Is there something wrong with a world in which everybody does lemma-lemma-theorem-appendix work? Of course. But the tactic used here is not particularly helpful, I think.
Barkley Rosser
May 17 2007 at 8:12pm
Well, I am discussing this issue in two places.
So, Pedro, 16 pages of appendices are a lot.
However, would you prefer that they put the
actual proofs in the bodies of the papers?
Would you prefer that they not provide the
proofs at all, perhaps offering a website for
those interested to go to read them?
Again, there is the matter of audiences here.
The issue is that you guys are walking in
as non-standard audiences for these kinds of
papers. You are in effect demanding that each
of them justify what they are doing to a
broader audience than the authors and editors
expect will be reading the papers. You may
not like lots of pages of proofs, but there
are readers of JET who want to see the darned
things.
As I noted over on MR, would you demand that
Southern Economic Journal, or, for that
matter, the Review of Austrian Economics,
insist that every article explain what would
be extremely basic concepts for the occasional
reader not sufficiently versed in them, e.g.
the basic law of supply and demand?
Also, again, not sure how long you guys
have been paying attention, but, really,
JET is a lot more readable than it used
to be, really.
Daniel Klein
May 17 2007 at 8:22pm
To Gabriel Mihalache:
First, I want to pick up on a passage that Bryan Caplan quoted from the paper.
Pedro and I wrote:
“Surely, the strongest defenders of model building would admit that the practice might go too far, into unjustifiable realms of creative writing. Economists will disagree about the “marginal conditions,” but every economist will affirm some marginal conditions, which imply the hazard of going too far. Every economist, therefore, should appreciate a watchdog effort, even if he objects to the watchdog’s test and results.”
May I assume that you agree?
If so, then I wonder where your marginal conditions are, or what they would look like. How far would “the top journal in economic theory” have to go for you to agree that most of what’s in it isn’t economic theory?
Or do you hold the view that economic theory is whatever elite academic economics happens to say it is?
Second, you write:
“Prof. Klein, why fail the authors of the paper [#15] whose summary is “We show the robustness of the Walrasian result obtained in models of bargaining in pairwise meetings.” on the “Theory of What?” point? . . . Should they have started with “This is a paper on prices.” to qualify?”
In the few hours I have to respond while the “Comments” field to Bryan’s post is still fresh, I have decided that it is not appropriate for me to try to answer challenges in this mode. I would rather do it in a more serious forum. Again, I encourage a scholarly piece submitted to EJW. Fred Foldvary handled the article, and he would handle follow-up and correspondence (ffoldvary@scu.edu).
If you write a reply, I encourage you to think about how Pedro and I would respond. In your post, do you mean to suggest that the article is a theory of how prices are determined? Would that be all prices, including the prices of pecans in Fairfax, Virginia? If so, what does that theory really tell us about the prices of pecans in Fairfax? (Or any other real-world prices you care to name.) What inadequacy does the new theory overcome? And how does it compare to other theories of pecan prices, including naïve or simple theories? If it does not cover pecan prices, which narrower class of prices constitutes the explanandum? Where does the article give empirical content to what that class is? Or even illustration?
In fact, I suspect that article #15’s real achievement is something like the following: Showing a simpler or less restrictive set of assumptions that deliver the same classes of equilibria as in previous models. The real question, then, is whether the new model really teaches us something better or more efficiently than the previous models. In that sense, it might shed light on price determination, as opposed to being merely model craftwork. But if there is some meaningful advance in understanding, it has eluded our repeated reading of the article. After all, reducing the number of assumptions is not necessarily desirable, if it makes the scheme harder to enter into and does not deliver any better intuition or understanding. A story might be more meaningful and more useful if its assumptions take intuitive form. As Robert Sugden says, what counts is meaningful credibility in the story that is set up to illuminate certain ideas, not simply a minimization of assumptions. The latter is the kind of thing that mathematicians, not economists, make primary.
Also, challenge a number of scorings, not just one. It would give us more to explore in finding where our differences really lie. Also, Pedro and I may have misjudged a few, of course. But if the “pass” rate goes up from 12 percent to 20 percent, it still means that most JET articles don’t qualify as theory.
Also, I would encourage you to say something about the “marginal conditions” issue (the first point above). Presumably, you have some marginal conditions, and the real issue is why we (Pedro and I) feel the appropriate conditions are on one side of current JET practice, and you (apparently) feel that they are on the other side.
To conchis:
You write: “The important issue [i]s whether those questions are able to be answered, not whether they actually were. That’s the issue that speaks to the value of any given instance of model building – as opposed to whether authors have sufficiently advertised the value of their modelling to non-specialists.”
You certainly have a point, but Pedro and I looked for mere indications of answers. We looked for mere illustration of the explanandum, mere claim of reason to care, mere claim of merit in the offered explanation. Is it unreasonable to demand that a theory article ought to give at least some brief consideration to the fundamental questions? Particularly in the current climate of doubt about model building? As Pedro and I show in the paper, we are not the only ones who suspect that JET and similar journals have gone too far.
BTW, what is your real name? Who are you? Where are you coming from on all this?
Signing off, Dan Klein
Barkley Rosser
May 17 2007 at 11:54pm
I realize that Dan Klein has signed off and
wants everybody to retire to EJW and Fred
Foldvary’s room there to continue the discussion.
But I would like to note one point.
So, it is easy to sneer at or dismiss a paper
that amounts to showing that a particular result
(or “class of equilibria outcomes” or whatever)
can be shown with fewer (or simpler) assumptions.
However, even if authors do not present examples
of the real world relevance (and I for one prefer
papers that do provide such examples, but then
I do not edit JET), sometimes it is the case that
there is something highlighted about real world
processes by clarifying as narrow a set of assumptions that will achieve a certain result.
More particularly, if one has found such a minimum set of assumptions, then the failure of one in the real world may well indicate or be associated with a failure of the outcome to hold in the real world as well. Thus, real world failures of particular axioms of preference from general equilibrium theory are associated with particular forms of behavior that are studied in the experiments of behavioral economists. One can easily go on.
Gabriel M.
May 18 2007 at 12:11am
Thank you for your reply!
I’m personally uncomfortable with the current level of technicality used for even the most banal or applied issues (Yes! Let’s use Measure Theory to get a real axiomatic approach to the business cycle!). That being said…
I have an aversion to your approach because, it seems to me, that you’re purposefully acting as if you don’t understand what these papers are about. But in fact, as PhDs, you are more than familiar with the technicalities and the existing traditions and schools of thought.
Readers of the JET do find meaning and insight in those damn proofs. And they’re familiar with the body of literature in the context of which those paper make sense.
I’m sure we can find definitions of “theory” according to which G.E. is not “theory”. But G.E. is theory to economists (I do think that Economics is what economists do, sadly.) — For those working within the Walrasian framework or on the Nash program, it would be nonsensical to restate the basic issues they try to address within every paper, even in the smallest contributions.
Re: “prices of pecans in Fairfax, Virginia”… I expect theory to elucidate the theoretical notion of price, that the conclusions would have an abstract and general character.
In that sense, the theoretical notion of price is about all prices, even those in Fairfax. Even so, there’s a wider, more philosophical problem on the relation between abstract thought and the world. But that’s probably not what we want to deal with here.
conchis
May 18 2007 at 5:53am
Dan says:
I don’t think it’s at all unreasonable to think that a theory article should ideally address these questions explicitly. What I do think is unreasonable is to demand that it do so in order to qualify as theory. There’s quite a difference between those two things, which I’m not sure your methodology adequately accounts for.
P.S. Why does it matter who I am?
Daniel Klein
May 18 2007 at 7:56am
Barkley writes:
“However, . . . sometimes it is the case that
there is something highlighted about real world
processes by clarifying as narrow a set of assumptions that will achieve a certain result.”
Sure. The thing to do would be identify what that something is in article #15.
It might be there. I’m not sure. But, also, it might not be.
Barkley Rosser
May 18 2007 at 1:09pm
Dan K.,
Getting close to closure here, I think we are.
Sure, I would prefer to see papers explicate clearly their relevance. However, their failure to do so does not therefore render them “not theory,” as I have discussed at greater length over on MR. Again, for the umpteenth time, they are usually presuming that readers know the contexts and potential apps, even if you and Pedro and I would prefer that they spell them out better.
So, you and Pedro have reasonable grounds to complain about the lack of clarity and explanation in these papers. You are on ground not readily defended when you go further to claim that these papers are “not theory” or perhaps “not about theory.” That just does not fly, and I think that is why you got such a sharp and aggrieved reaction, even if Bryan may think it was amusing, and of course as most of those doing the name-calling now realize, much of the initial reaction was also overly heated.
Barkley Rosser
May 18 2007 at 1:12pm
Of course another way to go would be to argue that abstract math is not the best way to do theory, and I think that really is the fish you are trying to fry. But that is also quite different from saying that formalistic-deductively mathematically abstract economic theory is not theory, unless it is accompanied by appropriate general explanations or discussions of potential real world applications.
Jim McClure
May 18 2007 at 3:57pm
JET is a journal that embraces long chains of deductive reasoning in the form of papers that have extended mathematical appendices in addition to text that is often more math than text. In 1920, Alfred Marshall indicated that he thought that there was no place in economics for these. In 1955, Don Gordon (cited in K&R) expanded upon Marshall’s idea with a hypothesis about the timelessness implicit in mathematical chains reducing a theory’s applicability the longer the chain. In 2005, Coelho and I operationalized Gordon’s hypothesis in an article in the Southern Economic Journal. Our evidence is that papers in the AER with longer mathematical chains, by our metrics, were less likely to themselves contain empirical results; and the long-chain-papers were also less likely to be cited by subsequent papers that contained any empirical work. The literature cited in the K&R paper is not unimportant to the persuasiveness of their argument.
Barkley Rosser
May 18 2007 at 4:53pm
Jim,
You may be right about citations in empirical work, but JET remains a very highly cited journal in economics, especially along impact-adjusted lines.
Jim McClure
May 18 2007 at 8:44pm
Barkely,
I don’t think that the “impact adjustment” you mentioned makes a delineation between cites by pubs about observational reality (with empirical results) and those that do not; therefore I don’t think that we have any disagreement.
Comments are closed.