23 Jul 2007

Peer Reviewers as 'Gatekeepers':
Barry Gewen of the New York Times Book Review

by Richard Belzer

in Peer Review

Scholarly peer review is a quality control exercise in which a journal editor decides which competing scientific papers should be published in the limited number of pages available. The number of scholarly journals has increased, in part, to accommodate the much greater (and rising) number of scholarly manuscripts that meet a broadly accepted minimum quality standard for publication. In science, publication is essential for having one's research taken seriously.

But some peer review activities are motivated by other considerations. Colloquy, the quarterly review published for alumni of Harvard's Graduate School of Arts and Sciences, has published a cover article on Barry Gewen, a GSAS alumnus (history of American civilization, 1972) who is a preview editor for the New York Times Book Review. For 17 years, he has decided which books in foreign policy, history, economics and current events are sent out for review. He describes his job as bing a "cultural gatekeeper." More...

29 Jun 2007

Distinguishing Risk Assessment from Risk Management:
Telling them apart can be hard

by Richard Belzer

in Information Quality, Peer Review, Regulatory Science

Experts in risk analysis often distinguish between risk assessment and risk management. But sometimes they don't, and that can leave the public confused about the difference.

Today's example is trichloroethylene, commonly called TCE. More...

16 May 2007

Journal Peer Review and Objectivity:
Corroboration of the problem from scientists themselves

by Richard Belzer

in Peer Review

We've posted on the "rebuttable presumption" of objectivity that the Office of Management and Budget wrote into its government-wide guidelines for information quality and peer review. The core of the problem is that scholarly journals do not use objectivity (as OMB has defined it) as a criterion for acceptance, or as a performance standard for peer reviewers. That means peer review is a poorly targeted (and perhaps completely unguided) way to ensure that federal agencies disseminate information that satisfies the objectivity criterion.

Other scientists are chiming in, with interesting perspectives on various underlying problems in journal peer review. A number of reforms have been suggested and are worth examining. Note that the list of identified problems does not include objectivity as OMB defined the term. Last year we noted that Nature is trying an "open peer review" model, one that uses blogging technology. We have not yet found an analysis of how it has performed.
More...

11 May 2007

Government-wide Information Quality Guidelines:
Does journal peer review achieve "adequate" objectivity?

by Richard Belzer

in Glossary, Information Quality, Peer Review

Federal guidelines require information disseminated by federal agencies to satisfy a few broad criteria, one of which is objectivity. These guidelines give a "rebuttable presumption" to scientific information published in scholarly journals.

• Do all scholarly journals transmit this rebuttable presumption of objectivity? What about scholarly journals that also have an advocacy mission?
• What is the burden of proof for mounting a successful rebuttal?

10 May 2007

When Science Becomes Advocacy:
Childhood obesity in Somerville, Massachusetts

by Richard Belzer

in Peer Review, Regulatory Policy

Wall Street Journal health columnist Tara Parker Pope has a Page One article about a recent program intended to reduce the incidence of childhood obesity in Somerville, Massachusetts. The occasion for the story is publication of a scientific review of the program.

Pope's article, and the journal article on which it is based, raise troubling questions about scientists and journalists behaving as advocates. It also exposes problems inherent to scientific peer review. More...

13 Mar 2007

Hype and Science
A new look at climate change policy and the science supporting it

by Richard Belzer

in Peer Review, Regulatory Policy, Regulatory Science

We've posted three times on different approaches to climate change, distinguishing the views of CO2 Pragmatists from CO2 Puritans. Whereas Pragmatists are open to market-based regulatory responses such as carbon taxes and tradable emission permits, Puritans are not. CO2 Hybrids are Pragmatists who are invested in technological remedies beyond what economic efficiency would achieve with correct market prices.

Most recently we implicitly defended Al Gore from criticism on both the right and left that he was insufficiently Puritan in his personal "carbon footprint," which appears to be rather large. We noted that the policy prescriptions on the web site for An Inconvenient Truth are consistent with CO2 Pragmatism. Thus, the strongest basis for criticism would arise if Gore did not actually behave as a CO2 Pragmatist by purchasing bona fide carbon offsets for the CO2 emissions he indirectly generates. We've found no evidence that he doesn't, and plenty of evidence that he can easily afford to do so.

In today's New York Times, William J. Broad identifies a different basis for criticism. The scientific claims about climate change in AIC are the subject of increasing controversy among climate change scientists.

26 Jan 2007

NAS Opens Up (a Bit) to Public Participation
Finally, the public can effectively comment on provisional committee appointments

by Richard Belzer

in Peer Review

The National Research Council, the operating arm of the Academy of Sciences, routinely conducts peer review of scientific issues. The experts it provisionally appoints to these review panels are subject to a 20 calendar day public comment period.

Historically, the public's actual ability to comment has been severely limited and in violation of the spirit (if not the letter) of the Federal Advisory Committee Act. NRC would post notice of a provisional committee appointment, but not alert the public that it had done so. Members of the public would have to know that a review was about to begin; the name of the board, committee or other unit under whose auspices the review was taking place; and possibly the name of the ad hoc committee. Then they would have to check the NAS web site every day.

For an organization that claims to be the premier scientific and technical body in the United States, this scheme is remarkably archaic. Today, one NRC division finally made it easy for the public to monitor its activities.

Information Quality, Peer Review, and Consumer Reports
Misleading child safety seat tests withdrawn

by Richard Belzer

in Information Quality, Peer Review, Regulatory Science

A recent story involving an erroneous test report about child safety seats published by Consumer Reports shows how information quality is not just a concern related to information disseminated by the federal government.

It also illustrates the value of genuinely independent peer review. Consumer Reports is published by Consumers Union, a nonprofit organization that does much more than product testing. It is an activist organization that routinely takes strong positions on a wide range of public policy issues, including child safety seats.

CU's activism creates an inherent conflict of interest with its product testing functions. Credibility as a product safety tester requires, at a minimum, an extraordinarily rigorous program of independent peer review. Currently, CU relies solely on internal peer review.

14 Sep 2006

Nature Adopts an Experiment on Open Peer Review

by Richard Belzer

in Information Quality, Peer Review, Regulatory Science

Nature, one of the world's most distinguished science journals, recently supplemented its existing anonymous peer review with an open peer review process. A short story about it appeared in a UPI report, a longer one in Thursday's Wall Street Journal (subscription required). Scholarly peer review traditionally has been anonymous and confidential. It's purpose has been to ascertain whether scientific research reports meet the minimum quality standard for the journal in question, and secondarily, to allocate scarce pages to the best science that is compatible with the journal's mission.

The effectiveness of this model has come under frequent criticism for failing to detect fraud or error -- both of which are different and much more demanding objectives. In public policy, the Office of Management and Budget's 2002 Information Quality Guidelines confer a rebuttable presumption of objectivity on science that has survived scholarly peer review. This is an odd space for scholarly journals to occupy because objectivity, as that term is defined in the OMB guidelines, rarely is the defined purpose of scholarly publication.

7 Sep 2006

October 9-12: Naphthalene State of the Science Symposium

by Richard Belzer

in Peer Review, Regulatory Science

What is the future of human health risk assessment? The Naphthalene State of the Science Symposium will gather distinguished researchers and experts and sort out a path forward.

EPA's Office of Research and Development has recently embarked on a significant reform to replace point estimates of risk with ranges described using the tools of uncertainty analysis. Naphthalene will be the first application of this new direction.

23 Aug 2006

Are Businesses More Likely to Produce Biased Research than Government Agencies or Nongovernmental Organizations?
Virginia offers an example

by Richard Belzer

in Information Quality, Peer Review

We have posted on the matter of whether scientists affiliated with industry should be automatically excluded from federal peer review panels irrespective of the scope, scale or significance of their affiliation. The Center for Science in the Public Interest says any financial tie (a term it does not clearly define) with any industry constitutes a "potential" conflict of interest, and any financial tie of any magnitude with any industry or group that might be directly affected constitutes a "direct" conflict of interest.

We expressed concern about the breath of these proposed automatic exclusions. CSPI's stated justification is that industry funding may result in bias, and one commenter on our post asserted that "the possibility of bias in industry funded work is greater than in other work since I believe that financial incentives are greater than non-financial ones." But neither CSPI (nor the anonymous commenter) appears to be concerned about financial ties with regulatory agencies, activist groups and their supporting foundations irrespective of scope, scale or significance. If CSPI's conflict of interest rule were to apply, only for-profit financial interests would be grounds for exclusion. Thus, CSPI would succeed in automatically excluding its opponents and thereby silencing views contrary to its own.

The focus on financial interest alone is misguided. Serious conflicts can occur because of political or policy interests without the exchange of money. An example from the Commonwealth of Virginia illustrates our point. In this case, an ostensibly neutral traffic study was managed by senior political aides to the governor. They apparently hid their involvement until exposed by an activist with an opposing view.

27 Jul 2006

NAS Report on Cancer Risks from Trichloroethylene (TCE)

by Richard Belzer

in Information Quality, Peer Review, Regulatory Policy, Regulatory Science

On July 27, the National Research Council of the National Academies released a report concluding that while additional research should be performed, enough information exists for the Environmental Protection Agency to complete its risk assessment of trichloroethylene. EPA's 2001 draft risk assessment was highly controversial. Similar to what has happened after EPA published other controversial draft risk assessments, several other federal agencies joined with EPA to sponsor the NRC review (in this case, the Departments of Defense and Energy and NASA).

25 Jul 2006

NAS Asks Affected Federal Agencies to Opine on OMB's Proposed Risk Assessment Bulletin

by Richard Belzer

in Information Quality, Peer Review, Regulatory Science

On June 28, staff of the National Academy of Sciences panel charged with reviewing OMB's proposed risk assessment guidance sent OMB staff a letter "regarding questions from the NRC Committee to Review the OMB Risk Assessment Bulletin for the federal agencies." The letter apparently included a list of questions that committee members wanted affected agencies to address.

NAS posted a reference to to this letter on the project web site but did not post a copy of the letter iitself. Our request to OMB for a copy of the letter was denied, so today we submitted a formal request for it under the Freedom of Information Act.

Neutral Source has obtained what we believe to be an accurate deception of the attachment.

21 Jul 2006

Senate Approves Requirement for Peer Review of Army Corps of Engineers Water Projects

by Richard Belzer

in Legislation, Peer Review, Regulatory Economics, Regulatory Science

On July 19, the Senate approved by roll call vote of 56-43 an amendment to the Water Resources Development Act of 2005 that would require peer review of water projects undertaken by the Army Corps of Engineers.
More...

19 Jul 2006

Should Industry Representatives Be Excluded from Federal Advisory Committees?

by Richard Belzer

in Peer Review, Regulatory Science

The Center for Science in the Public Interest is sponsoring a "Forum on Corporate Influence on Federal Advisory Committees," July 24 at the National Press Club in Washington. At the event, CSPI promises to release a report on whether National Academy of Sciences panels are "fair and balanced."

Although agency-directed peer review panels are federal advisory committees, most advisory committees are not peer review panels. They are comprised of representatives from a broad range of stakeholder groups, whose function is to provide policy advice based on the interests they represent. In short, advisory committee members are expected to have financial and other interests coincident with the interests they represent.

Why would it make sense to exclude industry stakeholders from federal advisory committees charged with giving policy advice?

More...