Journal Peer Review and Objectivity:
Corroboration of the problem from scientists themselves
16 May 2007 in Peer Review
We've posted on the "rebuttable presumption" of objectivity that the Office of Management and Budget wrote into its government-wide guidelines for information quality and peer review. The core of the problem is that scholarly journals do not use objectivity (as OMB has defined it) as a criterion for acceptance, or as a performance standard for peer reviewers. That means peer review is a poorly targeted (and perhaps completely unguided) way to ensure that federal agencies disseminate information that satisfies the objectivity criterion.
Other scientists are chiming in, with interesting perspectives on various underlying problems in journal peer review. A number of reforms have been suggested and are worth examining. Note that the list of identified problems does not include objectivity as OMB defined the term. Last year we noted that Nature is trying an "open peer review" model, one that uses blogging technology. We have not yet found an analysis of how it has performed.
Janet Stemwedel identifies several problems in journal peer review, including:
- the ability of scholarly rivals acting as peer reviewers to sabotage research worthy of publication
- a lack of genuine anonymity, especially as subfields become increasingly abstruse
- lack of incentives to serve as peer reviewers, especially without either financial or nonfinancial compensation
To get incentives right, the first place to start is with a careful diagnosis of the problem that better incentives are supposed to solve. For example, scholarly rivalry isn't necessarily a bad thing -- indeed, it's probably the best way to ensure a thorough and rigorous review and to prevent substandard research from being published. The problem arises when journal editors fail to take account of scholarly rivalry when they decide how to use this information. Rivals who recommend publication are acting against type and should be accorded great weight in the review process. On the other hand, when rivals recommend rejection editors should carefully take rivalry into account. An easy way for journal editors to improve incentives is to set higher performance standards when scholarly rivals serve as peer reviewers.
As great a problem that scholarly rivalry might be as a deterrent to the publication of deserving, high-quality manuscripts, it's easy to forget that scholarly harmony in peer review can lead to the publication of undeserving, low-quality manuscripts. No scholar whose substandard work is accepted will ever object, but as long as journal pages are scarce the publication of substandard papers will crowd out more deserving research. But the identities of the scholars crowded-out will not be known. That they are not known does not make them less real.
We point out at this point that this is a clear example of a market failure. Yet there is no reason to believe that government could do anything to remedy it. The existence of a market failure says nothing at all about whether government regulation could solve it.
This observation provides a natural segue to the problem of limited anonymity, for which there is no simple solution. Anonymity is a useful tool for encouraging candor, but the ability to provide candid review does not always lead reviewers to actually be candid. In some cases, reviewers promised anonymity may have reason to distrust the confidentiality of the editorial process. Moreover, anonymity creates its own set of problems, most notably a way to evade being held accountable for a review that is unduly harsh or generous. Reviewers most likely to want to evade accountability are scholarly rivals and scholarly harmonics. Perhaps journal editors might consider experimenting with systems in which reviews by rivals and harmonics are purposefully non-anonymous, with anonymity reserved to those reviewers who are genuinely independent or have a proven track record of rigor and fairness.
The lack of incentive to perform peer review comes down to finances, and the simplest solution is to pay peer reviewers honoraria for their service. In our experience, honoraria on the order of $1,000 are sufficient to motivate scholars to take the job seriously and make it a higher priority. Journal editors will ask where the cash will come from to pay these sums, but a careful review of the full costs of running a journal likely will reveal that the cost of honoraria is small compared to other costs, and to the benefits of improved quality and timeliness that honoraria would deliver.
