Government-wide Information Quality Guidelines:
Does journal peer review achieve "adequate" objectivity?
11 May 2007 in Information Quality, Peer Review, Glossary
Federal guidelines require information disseminated by federal agencies to satisfy a few broad criteria, one of which is objectivity. These guidelines give a "rebuttable presumption" to scientific information published in scholarly journals.
- Do all scholarly journals transmit this rebuttable presumption of objectivity? What about scholarly journals that also have an advocacy mission?
- What is the burden of proof for mounting a successful rebuttal?
Information Quality Guidelines published by the Office of Management and Budget create a "rebuttable presumption" that articles published in scholarly journals satisfy the criterion of objectivity.
The Presumption Favoring Peer-Reviewed Information. As a general matter, in the scientific and research context, we regard technical information that has been subjected to formal, independent, external peer review as presumptively objective. As the guidelines state in paragraph V.3.b.i: ‘‘If data and analytic results have been subjected to formal, independent, external peer review, the information may generally be presumed to be of acceptable objectivity.’’ An example of a formal, independent, external peer review is the review process used by scientific journals (p. 8454, emphasis added).
Common sense suggests that where a journal has an advocacy purpose this rebuttable presumption does not apply because the journal is not scientific. Otherwise, OMB would give equal weight to peer reviewed journals irrespective of whether science or advocacy was the journal's underlying purpose. But that would not solve the problem posed by journals that interweave policy and science such that the boundaries between them are difficult or impossible to locate.
Given OMB's language, the only plausible remedy is to make sure the presumption of objectivity is relatively easy to rebut in cases where advocacy can be shown to dominate. But OMB's guidelines are silent with respect to the burden of proof.
OMB's peer review guidelines make up for this defect, but only in part. They make clear that journal peer review may have been adequate, but it is not per se sufficient:
[P]rior peer review and publication is not by itself sufficient grounds for determining that no further [peer] review is necessary.
Problems remain insofar as agencies are required to ensure that journal peer review, was "adequate" before they rely on it:Adequacy of Prior Peer Review: For information subject to this section of the Bulletin, agencies need not have further peer review conducted on information that has already been subjected to adequate peer review. In determining whether prior peer review is adequate, agencies shall give due consideration to the novelty and complexity of the science to be reviewed, the importance of the information to decision making, the extent of prior peer reviews, and the expected benefits and costs of additional review (Section II(2)).
OMB does not define "adequate," which leaves us with a rebuttable presumption of uncertain magnitude and unspecified procedures and standards for overcoming it. Furthermore, federal agencies have no special expertise in assessing the adequacy of peer reviews conducted by scholarly journals over which they have neither control, influence, or special access. (The journal Environmental Health Perspectives may be a good counterexample. EHP is published by the National Institute of Environmental Health Sciences, a division of the National Institutes of Health and a subunit of the Department of Health and Human Services.)
So with limited exceptions such as scientific information published in EHP, federal agencies cannot adhere to this aspect of OMB's peer review guidelines because they cannot credibly assess the adequacy of journal peer review. What they can do is take into account journals' mission statements, published peer review policies, and disclosure practices.
Below we list some criteria for discerning the extent to which journal peer review deserves a presumption of objectivity. Of course, in all cases, substantial evidence that scientific information in a journal isn't objective, the presumption should be considered successfully rebutted.
|
Factors to Consider in Evaluating Whether Journal Peer Review
Justifies a Presumption of Objectivity |
|||
| Decreased Confidence in Scientific Objectivity |
Increases Confidence in Scientific Objectivity |
||
| Journal Mission |
|
|
|
| Policy Advocacy |
|
|
|
| Editors and Editorial Board |
|
|
|
| Letters, rebuttals and other criticism |
|
|
|
| Intellectual Diversity |
|
|
|
| Data, models and methods |
|
|
|
| Inferences |
|
|
|
| Policy recommendations |
|
|
|
