OMB's Proposed Risk Assessment Bulletin: Would the NAS Perchlorate Review Comply?
8 Jun 2006 in Regulatory Science, Regulatory Policy, Information Quality, Peer Review
Are OMB's proposed risk assessment guidelines so demanding that even
the hallowed National Academy of Sciences wouldn't be able to comply? This question came up at a May 24 congressional briefing. We sort out the arguments pro and con.
In remarks delivered at the first public meeting of the National Academy of Sciences ad hoc committee reviewing the Office of Management and Budget's proposed risk assessment guidance on May 22, former Office of Information and Regulatory Affairs Administrator John Graham cited EPA's perchlorate risk assessment (archived here) as one of three examples he said illustrated why the OMB guidance was needed. Graham used this example to illustrate a category of risk assessments that are neither regulatory nor deregulatory but which have significant real-world effects. He implied that if the OMB guidelines had been in place and adhered to by EPA, the scientific controversy that arose would have been avoided.
We don't think so. Much of the scientific controversy about perchlorate centered on the relative quality of underlying animal and human data. Including the technical innovations called for by OMB, such as quantitative variability and uncertainty analysis, probably would have had little effect unless quality defects in certain underlying data had been identified before the risk assessment had been performed. Applying innovative risk assessment methods to substandard data does not yield higher quality estimates of risk.
Even though it occurred at the end of the process, the NAS review (free PDF available) primarily concerned data quality and model validity. If it had been available before EPA's draft toxicological profile had been written, the Agency's risk assessment would have been very different. Scientific controversy would have been avoided because it was the dissemination of the 2002 draft risk assessment, which relied on substandard data as well as a flawed risk model, that created the scientific controversy. More sophisticated risk assessment methods would not have helped.
In his public remarks at the NAS meeting, Graham said the NAS review was the only way to mediate the resulting interagency conflict once it had occurred. He noted that the NAS panel drew very different conclusions about the science than did EPA, clearly implying that these differences consisted of fundamental scientific errors in the EPA draft risk assessment. Graham praised the NAS panel for its alternative risk assessment and asserted that EPA's subsequent adoption of the NAS alternative was scientifically sensible and appropriate.
Graham elaborated on the perchlorate case at a May 24 congressional briefing jointly sponsored by the American Chemical Society and the Society for Risk Analysis. During this briefing, questions arose whether OMB's proposed standards are so strict that even the NAS perchlorate review would have failed to conform to the proposed guidance. Assuming the trade press account of the briefing in a story published May 30 by Risk Policy Report (subscription required) is accurate, several different issues must be sorted out.We address these questions in order below.
Did EPA's 2002 Draft Toxicological Profile for Perchlorate Adhere to OMB's Proposed Risk Assessment Guidelines?
At the NAS public meeting, Graham clearly implied that it did not. But he did not identify which specific provisions of the draft OMB risk assessment guidelines EPA's 2002 draft toxicological profile had failed to fulfill. Rather, he drew this inference from the observation that the NAS perchlorate committee rejected numerous scientific elements of the EPA risk assessment.
According to Risk Policy Report, Graham provided more details concerning the defects in EPA's draft perchlorate risk assessment at the May 24 congressional briefing:
Graham argued that if EPA followed OMB’s proposed requirements to provide “central” risk estimates, in addition to what he claims are EPA’s conventional “worst-case” scenarios, and faced stricter mandates for defining the adversity of the health effects, this would have dampened the infighting that broke out among federal agencies about perchlorate’s risks in 2002-03.If this account is complete, then the principal defects in EPA's draft perchlorate risk assessment were its reliance on a "worst case" risk model and its failure to take account of more plausible models that yielded negligible risk estimates. Central among the elements of this "worst-case" model were three crucial scientific inferences upon which the Agency's draft risk assessment depended:
- "Iodide uptake inhibition by the thyroid gland is (at least implicitly) an adverse effect."
- "Iodide uptake inhibition is the first of a cascading series of effects that leads to neurodevelopmental injury in the developing fetus."
- "Perchlorate exposure in utero can predispose humans to cancer later in life."
Inhibition of iodide uptake by the thyroid clearly is not an adverse effect; however, if it does not occur, there is no progression to adverse health effects (emphasis in original) (pp. 166-167).Similarly, the NAS committee rejected the notion that iodide uptake inhibition was first of a cascade of effects leading to harm:
The committee emphasizes that inhibition of iodide uptake by the thyroid has been the only consistently documented effect of perchlorate exposure in humans. The continuum of possible effects of iodide uptake inhibition caused by perchlorate exposure is only proposed and has not been demonstrated in humans exposed to perchlorate... More important, the outcomes at the end of the continuum are not inevitable consequences of perchlorate exposure (p. 165).As for cancer, the committee was equally dismissive. It endorsed a 2001 report by the International Agency for Research on Cancer, which states that substances which cause certain tumors in rodents are not human carcinogens:
"Agents that induce thyroid follicular cell tumors in rodents by interfering with thyroid hormone homeostasis, can with some exceptions, notability the sulfonamides, also interfere with thyroid hormone homeostasis in humans if given at a sufficient dose for a sufficient time. These agents can be assumed not to be carcinogenic in humans at concentrations that do not lead to alterations in thyroid hormone homeostasis" (pp. 144-145, quoting IARC).The committee concluded, in strong words for scientists, "It is unlikely that perchlorate poses a risk of thyroid cancer in humans. (p. 145)
Considering the breadth of error contained here, there is little doubt that if just these modeling errors had not been made it "would have dampened the infighting that broke out."
Would Adherence to OMB's Proposed Risk Assessment Guidelines Have Assured This Outcome?
It's not clear, however, that interagency conflict would have been avoided if only EPA had considered alternative risk models. According to the NAS committee, the EPA risk model was simply wrong. Merely considering alternative models, as the draft OMB risk assessment guidelines seem to require, is not a sufficient remedy for scientific error.
At least as important as the validity of the risk model was the quality of scientific data EPA had relied upon. An important element of the interagency dispute concerned the quality of certain animal data. The NAS rejected these data, concluding that the evidence was "inadequate to determine whether or not a causal relationship exists between maternal perchlorate exposure and pup neurodevelopmental abnormalities" (p. 89).
In principle, adherence to the high information quality standards OMB established for so-called "influential information" would have detected data deficiencies and prevented them from becoming the foundation of a risk assessment. In practice, however, this is unlikely to occur unless data are subjected to an information quality peer review after the literature review is complete but well before a risk assessment is written. Early peer review of data to ensure that they meet applicable information quality standards would have been necessary to avoid the interagency controversy about perchlorate. Instead, this data quality review occurred at the very end of a highly contentious process, and by an overly expensive and time-consuming tool -- a NAS review. Meanwhile, OMB's proposed risk assessment guidelines include no language ensuring that data satisfy applicable information quality standards before they are used in risk assessment.
Did the NAS Perchlorate Review Adhere to OMB's Proposed Risk Assessment Guidelines?
According to Risk Policy Report, congressional staff asserted that the NAS review, which Graham praised, did not adhere to the provisions of OMB's proposed risk assessment guidance:
But staff from the House Science Committee criticized Graham’s claim, saying the NAS review may not meet the bulletin’s objectives and OMB may be establishing unrealistic requirements for EPA risk reviews.
The staffers at the meeting questioned whether risk assessments performed even by leading experts would conform to the strict standards set out in the proposed bulletin. “Would the NAS perchlorate review have conformed with the OMB bulletin?” queried one majority [Republican] science panel staffer.
Graham did not respond at the time, but when asked in a later interview, said, “My guess would be yes with one exception: the NAS panel did not prepare a probability analysis or quantitative uncertainty analysis. However, one could argue that such an analysis was not ‘appropriate’ in that case. Recall the draft OMB guidance permits agencies to bypass analytic requirements when they are not appropriate in a specific situation.”
The answer depends on which part of the NAS review the staffer was asking about. Most of the report is not a risk assessment at all, but a peer review of EPA's 2002 draft toxicological profile and the scientific studies underlying it. The point of peer review is to assess the quality of a risk assessment, and in general it's inappropriate to subject peer reviews to the same performance standards as the documents they examine. If ever there was a process reform guaranteed to result in "paralysis by analysis," that would be it.
However, Chapter 5 of the NAS report was not a peer review. It was a risk assessment, and there is hardly any question that it did not adhere to OMB's proposed risk assessment guidelines. Indeed, it did not adhere to even the most fundamental principles of OMB's 2002 information quality guidelines or, for that matter, any previously known risk assessment method. The committee derived a recommended reference dose using a No Observed Effect Level as the point of departure (instead of a No Observed Adverse Effect Level), then divided that value by 10. The resulting value is more than 570 times less than the NOAEL identified by the NAS committee; conventional practice would have limited this adjustment to a factor of 10. There is no support for Graham's apparent assertion that the absence of a probability analysis or quantitative uncertainty analysis is all that separated the NAS committee's risk assessment from the proposed OMB risk assessment guidelines. These technical innovations are at best secondary matters, because the NAS committee's recommended reference dose was driven by precautionary policy considerations -- not science.
In his presentation to the NAS committee reviewing the proposed OMB risk assessment guidance, Graham praised the outcome of the NAS perchlorate committee's work, apparently because it solved a difficult interagency conflict and political problem. But he was silent about the propriety of the methods they used and was asked no questions about it by committee members. This discrepancy reinforces concerns among some opponents of the proposed OMB risk assessment guidance that the true test of whether a risk assessment adheres to the proposed standards is whether it supports someone's desired policy outcome.
What Lessons Should Be Learned from the Perchlorate Case?
That would be an unfortunate lesson to draw from the perchlorate case. A more accurate story is that the proposed OMB risk assessment guidelines are reasonable expectations for highly significant human health risk assessments. Agencies should not emphasize "worst case" risk models, and if the stakes are high enough, conduct quantitative variability and uncertainty analyses. But neither the EPA draft risk assessment nor the NAS committee ad hoc risk assessment adhered to these standards.
As a private organization, NAS would be exempt from any OMB risk assessment standards. The problem for Graham is that EPA adopted by reference the NAS risk assessment, and he supported that action. But EPA could not have done so if the OMB risk assessment guidelines had been in place, and were responsibly applied.
