Climate Change v. Scientific Method:
Emails suggest a serious failure in peer review
23 Nov 2009 in Regulatory Science, Peer Review
Last week, unknown hackers broke into the computer at the University of East Anglia's (UK) Climate Research Unit, downloaded a trove of emails and other documents, then posted them on the web for all to see.
Peer review is the process by which scholars determine whether scientific inquiry meets the minimum quality standard for publication. It's also the process governments use to secure the endorsement by scientists of public policy decisions. The Intergovernmental Panel on Climate Change is both, and more; it performs both scholarly and governmental peer review functions, and it also proposes public policy actions, so it's a stakeholder in the climate change debate.
Both peer review processes have known problems. In scholarship, the minimum standard for publication varies greatly across journals and it is susceptible to the normal array of human vices and institutional pathologies. Still, there are thousands of journals and no one's permission is required to start one, so peer review is rarely, if ever, a barrier to the publication of high quality science. (The main problem in scholarly journals is that the peer review procedures they follow do not prevent a lot of low-quality work from being published. Thus, to say that a scientific work has been peer reviewed does not mean it has high quality.)
Government peer review is very different, and it suffers these pathologies and more. In particular, government peer review panels are organized to ratify the accuracy of government agency work products. Peer reviewers are selected by the authoring agency, or by an external firm that works on contact for the authoring agency and thus has a business case for selecting reviewers that, at a minimum, do not cause the client heartburn. Finally, government peer reviewers rarely are permitted to do what peer reviewers for a scholarly journal do all the time: say "No."
The e-mails and other documents disclosed by the hackers suggest that in the case of global climate change, both types of peer review have failed. Scientists are shown conspiring to exclude from both scholarly publication and governmental peer review competing scientists that they dislike and scientific studies with which they disagree. (In the days to come it will become clearer how successful they were.)
Keith Johnson of the Wall Street Journal summarizes:
The scientific community is buzzing over thousands of emails and documents -- posted on the Internet last week after being hacked from a prominent climate-change research center -- that some say raise ethical questions about a group of scientists who contend humans are responsible for global warming.
The correspondence between dozens of climate-change researchers, including many in the U.S., illustrates bitter feelings among those who believe human activities cause global warming toward rivals who argue that the link between humans and climate change remains uncertain.
Some emails also refer to efforts by scientists who believe man is causing global warming to exclude contrary views from important scientific publications.
---
A partial review of the hacked material suggests there was an effort at East Anglia, which houses an important center of global climate research, to shut out dissenters and their points of view.
In the emails, which date to 1996, researchers in the U.S. and the U.K. repeatedly take issue with climate research at odds with their own findings. In some cases, they discuss ways to rebut what they call "disinformation" using new articles in scientific journals or popular Web sites.
The emails include discussions of apparent efforts to make sure that reports from the Intergovernmental Panel on Climate Change, a United Nations group that monitors climate science, include their own views and exclude others. In addition, emails show that climate scientists declined to make their data available to scientists whose views they disagreed with.
East Anglia University's response is illuminating. The University asserts that the hackers disclosed material selectively:
The selective publication of some stolen emails and other papers taken out of context is mischievous and cannot be considered a genuine attempt to engage with this issue in a responsible way.
Whether the hackers motives are mischievous or patriotic is immaterial to the question whether the information disclosed is true (which the University does not deny) or biased due to selectivity (which the University claims). But the University declines to provide any means for scientifically testing this latter claim, and this raises questions about its own motives.
With regard to the substance of the climate change issue, the University attempts to use peer review as an appeal to authority:
The material published relates to the work of our globally-respected Climatic Research Unit (CRU) and other scientists around the world. CRU's published research is, and has always been, fully peer-reviewed by the relevant journals, and is one strand of research underpinning the strong consensus that human activity is affecting the world's climate in ways that are potentially dangerous.
This defense is ironic because what the newly disclosed emails show is a concerted effort to control the scientific and governmental peer review processes for the purpose of excluding some scientists and evidence for nonscientific reasons. Moreover, the fact that University researchers publish in peer reviewed journals does not address publication bias arising from the selective disclosure of scientific results obtained by University researchers. In short, critics can only examine what these researchers choose to submit.
The University's appeal to the authority of peer review also is confounded by two nonscientific matters. The University makes an explicit claim that a "consensus" exists about a hybrid statement that contains both scientific content ("human activity is affecting the world's climate") and nonscientific content ("in ways that are potentially dangerous"). Science progresses by continuous learning, which often requires that known "truths" be tested in accordance to the rules of science. The search for consensus, however, can quickly deteriorate into a political activity undertaken to secure favorable policy decisions by governments.
Scientific peer review can yield agreement, and agreement can be interpreted as provisional consensus. It is most legitimate when peer reviewers reach similar conclusions independently from different perspectives using different data. The less independent peer reviewers are, however, the less their agreement can be construed as a genuine consensus. When peer reviewers lack independence, and their agreement was predictable ex ante, any claim that a consensus exists is misleading. When they reach agreement by excluding data they dislike or persons with whom they disagree, claims that a scientific consensus exist are refuted under the normal rules of science, which rejects results that are the product of selection bias.
