Peer Review as a Reputation System
By Arnold Kling
Many people reacted to my take-down of peer review, and at least one commenter asked, “Can you come up with something better?”
Easily. The generic problem is something called “reputation systems.” For example, Google’s page-rank formula is a reputation system. If a particular page has the keywords you are looking for and a lot of links, it shows up at the top of Google’s search results,.
E-bay has a reputation system. Participants give ratings to transactions, and you can see the average rating of someone with whom you are about to transact business.
Peer review is a reputation system. It is closely tied to another reputation system, called tenure. Which is closely tied to another reputation system, called academic hiring and promotion.
I have issues with all of those reputation systems. Keep in mind that this could all just be sour grapes on my part.
I think that academic hiring and promotion create too much concentrated power among just a few grad schools and a few people at those schools. Part of the reason that macro is so desolate right now is that you can trace just about every prominent macroeconomist today to about half a dozen professors who were prominent at MIT, Chicago, and Minnesota between 1975 and 1985. That would be fine if everyone were mining an area that is rich in gold. But they were all mining an area that was devoid of gold.
I think that the focus on publications for tenure does not promote wisdom. A lot of tenured professors are clever without being wise. In fact, there is no way for wisdom to enter into the tenure equation. But if I look back on my own education, I place a very high value on the few professors who had wisdom and very low value on a lot of professors who had nothing but cleverness.
All that said, I think that the average outcome of hiring and tenure decisions is quite good. Lots of individual injustices, but the overall result tends to be reasonable.
However, of all the reputation systems at work in academia, I think that peer review for journals is the worst. With all of the modern tools available for disseminating ideas, commenting on ideas, rating ideas, citing ideas, and doing tabulations of all the ratings and citations, it is easy to come up with better systems.
For example, suppose I do some empirical research and report the results in a paper on line. Other people can rate my paper in terms of reliability, importance, or other criteria. Those ratings can be much more reliable than peer review. The system would require incentives for people to be honest. It probably would require a reviewer rating system so that the reviews of more reliable critics are weighted more highly than the reviews of less reliable critics.
If academics were a free market economy, I would expect peer-reviewed journal articles to die quickly and be replaced by alternative reputation systems. But the current system advantages incumbents enormously, and there is no reason for them to give up that advantage.
The hard part is not coming up with a better reputation system than peer-reviewed journals. The hard part is getting adoption of a system when the people who would have to adopt it have no incentive to improve the system and every incentive to leave it as it is.