Information Quality and Peer Review:
Are disclaimers in draft documents effective?
29 May 2008 in Information Quality
Since 2002, federal information quality guidelines have required agencies to avoid disseminating scientific information that is not objective, and to have effective administrative systems for managing requests for the correction of information that a petitioner believes is incorrect. The burden of proof of error rests with the petitioner.
All information that is "disseminated" is covered by these rules, but information that is made public solely for the purpose of scientific peer review or public comment is exempt from the definition -- provided that it is accompanied with a specified disclaimer (p. 8):
“THIS INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF PRE-DISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY [THE AGENCY] AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY AGENCY DETERMINATION OR POLICY.”
The purpose of this disclaimer is to deter people from relying on draft documents. An empirical question is whether the prescribed language is strong enough.
An interesting test case has arisen with respect to the industrial chemical bisphenol A (BPA).
BPA is used in the manufacture of polycarbonate plastics, resins and other products. Recently, the NIH's Center for the Evaluation of Risks to Human Reproduction (CERHR) and National Toxicology Program (NTP) published a "Draft Brief" that proposes to make certain apparently scientific statements about the human health risks posed by BPA based on a literature review:
- "[T]here is some concern for neural and behavioral effects in fetuses, infants, and children at current human exposures.
- "[There is] negligible concern that exposure of pregnant women to bisphenol A will result in fetal or neonatal mortality, birth defects or reduced birth weight and growth in their offspring."
- "[T]here is negligible concern that exposure to bisphenol A causes reproductive effects in non-occupationally exposed adults and minimal concern for workers exposed to higher levels in occupational settings.
The five levels of concern used by NTP and the CERHR expert panels are, from highest to lowest:
Serious Concern
Concern
Some Concern
Minimal Concern
Negligible Concern
The NTP or panel may express "serious concern" if a substance is found to cause reproductive or developmental effects in humans or if human exposures are similar to doses that clearly cause adverse developmental or reproductive effects in laboratory animals. At the opposite extreme, a conclusion of "negligible concern" may be warranted when the available data provide good evidence that the substance under evaluation is not a reproductive or developmental toxicant or human exposures are very low compared to dose levels that cause adverse effects in laboratory animals.
Serious Concern
Concern
Some Concern
Minimal Concern
Negligible Concern
The NTP or panel may express "serious concern" if a substance is found to cause reproductive or developmental effects in humans or if human exposures are similar to doses that clearly cause adverse developmental or reproductive effects in laboratory animals. At the opposite extreme, a conclusion of "negligible concern" may be warranted when the available data provide good evidence that the substance under evaluation is not a reproductive or developmental toxicant or human exposures are very low compared to dose levels that cause adverse effects in laboratory animals.
"Negligible" concern is the lowest level on this scale, and thus it must encompass zero risk. Nevertheless, even a careful reader cannot discern what likelihood of risk is large enough to bump a chemical up to "minimal concern," or where the threshold separating "minimal" from "some" concern is located. Nevertheless, each statement is apparently intended to be scientific, and thus it is covered by NIH's information quality guidelines and must be supported with sound scientific evidence rather than mere opinion. Also, the NIH guidelines require that these statements be scientifically objective -- that is, not motivated or influenced by anyone's policy judgment about whether bisphenol A ought (or ought not) to be regulated in some manner.
Prior to publishing this Draft Brief for public comment, CERHR obtained a report from an expert panel that was charged with reviewing the science in accordance with a published set of guidelines. However, these guidelines do not mention information quality or any of its established principles and criteria. Thus, it seems highly unlikely that the panel adhered to them.
Moreover, there is scant evidence that NTP adhered to its own information quality guidelines, either. The Draft Brief does not even mention the subject. The only place CERHR visibly implemented its information quality guidelines is in the disclaimer:
This DRAFT NTP Brief is distributed solely for the purpose of public comment and pre-dissemination peer review. It should not be construed to represent final NTP determination or policy.
How effectively did this disclaimer deter third parties from interpreting the Draft Brief as both a final work product and an objective statement of the science? Apparently, it was not very effective. First, a Google search for {BPA and (NTP or "National Toxicology Program")} during the past month yields over 13,600 hits.
Second, the Washington Post and several other news outlets today reported that an Arkansas woman filed suit in US District Court for Connecticut against Playtex (and all other entities that produced, manufactured, and/or otherwise distributed polycarbonate plastic bottle products). The lawsuit seeks nationwide class action status and, therefore, potentially billions of dollars in damages.For the plaintiff's claim BPA that has caused injury, a key scientific authority cited by the plaintiff is the NTP Draft Brief. If courts treat draft government reports as acceptable evidence, then the default disclaimer would be proved insufficient.
We have not yet reviewed the dozens of public comments on the Draft Brief to determine whether anyone challenged it on information quality grounds. Previously, the Environmental Working Group, an advocacy organization that opposes BPA, has complained about what it perceives as insufficient objectivity in the preparation of prior reports. Because objectivity is the core principle in information quality, presumably a consensus exists among all interested third parties that CERHR and NTP must ensure that the final version of the Brief must adhere to these principles.
