Distinguishing Risk Assessment from Risk Management:
Telling them apart can be hard
29 Jun 2007 in Regulatory Science, Information Quality, Peer Review
Experts in risk analysis often distinguish between risk assessment and risk management. But sometimes they don't, and that can leave the public confused about the difference.
Today's example is trichloroethylene, commonly called TCE.
Sen. Charles Schumer held a conference call with reporters and environmentalists to complain that the Environmental Protection Agency is taking too long to finalize a new standard for TCE. This is an issue in Upstate New York and Long Island where there are properties contaminated with the chemical, and residents want to know how much exposure is safe. The conference call was reported by Steve Orr of the Rochester Democrat and Chronicle, Eric Monnat of the Herkimer Evening Telegram, and Maruxz Relano and William Murphy of Newsday. A transcript appears not to be available.
The reporters indicate that Schumer berated the Environmental Protection Agency for failing to finalize a regulatory standard that it proposed in 2001. However, EPA does not have a standard to finalize. In fact, it already has a regulatory standard -- 5 ppb in drinking water- -which virtually all public water supply systems are required to meet.
DRAFT VS. FINAL RISK ASSESSMENTS
What EPA did in 2001 was publish a draft risk assessment. The purpose of the risk assessment was to take account of new science published sine 1987, and the reason for publishing it was to obtain scientific peer review. That it was a draft document is evident from the disclaimer printed on the title page:
THIS DOCUMENT IS A PRELIMINARY DRAFT. It has not been formally released by the U.S. Environmental Protection Agency and should not at this stage be construed to represent Agency policy. It is being circulated for comment on its technical merit and policy implications.
This disclaimer is neither as clear nor as prominent as it could be. Clarity is diminished because the wording is tentative; it has not been "formally" released, and it's not Agency policy "at this stage." It's not prominent because a reader need only miss the cover page to miss it entirely. Consequently, the public is routinely confused about the status of draft documents distributed for peer review and final documents disseminated to inform regulatory decision-making. When a disclaimer is clear and prominent, readers may misinterpret draft documents despite an agency's best efforts to deter it. But when an agency's disclaimer is vague and hidden, misunderstanding is routine.
For this reason, peer review guidelines issued by the Office of Management and Budget in 2005 require that disclaimers be worded more strongly:
"This information is distributed solely for the purpose of pre-dissemination peer review under applicable information quality guidelines. It has not been formally disseminated by [the agency]. It does not represent and should not be construed to represent any agency determination or policy."
Though it stopped short of requiring it, OMB itself has adopted the practice of publishing the disclaimer as a footer on each page of draft documents that have been distributed for peer review or public comment. This has been easy to do with conventional word processing technology for more than 20 years, so it's not clear why OMB had to issue guidance on the matter, or why agencies still fail to follow it. (On April 20, 2007, for example, EPA published for external peer review a draft risk assessment for 1,1,1-trichloroethane. OMB's required disclaimer is printed, but only on the cover page.)
The 2001 draft risk assessment for TCE was controversial from the outset. EPA had commissioned a set of state-of-the-science issue papers prepared by a diverse group of scientists, and the Agency had committed to use papers as the scientific foundation of the draft risk assessment. When it was published in 2001, the draft risk assessment apparently violated this commitment. Six issue paper authors took the highly unusual step of sending a letter to EPA administrator Christies Todd Whitman stating that while "the agency has widely publicized its intent to rely on this set of reviews and analyses by outside experts," it had adopted interpretations of the science with which they profoundly disagreed. "[W]e wish to make clear that there are a number of findings and conclusions about trichloroethylene in the EPA [draft risk assessment] with which we as state-of-the-science paper authors do not agree and which we would not endorse." (Their letter is not on the EPA web site.)
This dispute set the tone for EPA's peer review, which neither reduced nor resolved (and may have exacerbated) the scientific controversy. Ultimately, the scientific issues were punted to an ad hoc committee of the National Academies of Science to resolve. (The NAS is an honorary institution; its members generally do not actually do anything upon election. The NAS has an operating arm called the National Research Council, which is made up of a thousand hired staff members who organize ad hoc reviews when federal agencies ask [and pay] them to perform them.)
HOW RISK ASSESSMENT AND RISK MANAGEMENT DIFFER
"Risk assessment" is a scientific process of measuring or estimating risk; it should be objective. "Risk management" is a policy-based process used to decide what to do about it; it can't be objective. Science (and scientists) can help inform this choice, but science (and scientists) cannot decide it. Their expertise in scientific questions does not magically create expertise in the wide array of factors that regulatory decision-makers must take into account, not does scientific expertise equate with wisdom. Wisdom is not a scientific field for which universities award Ph.D. degrees.
Sometimes, risk assessment is used to inform risk assessment and sometimes it's not. Regulatory agencies are bound to follow the guidance given by statute. Statutes vary in what guidance they give. Some statutes, like parts of the Clean Air Act and the Clean Water Act, require regulatory agencies to establish regulatory standards based on "best" or "best performing" technology. Risk assessment -- the objective estimation of how much risk of what variety accompanies a fixed level and duration of exposure -- is immaterial to the decision. All that matters is what clever engineers can achieve.
Other statutes require regulatory agencies to based their decisions on risk, but in almost all cases they do not provide much insight about what the head of the agency is supposed to do with this information. Statutory instructions vary from setting standards that provide an "ample" margin of safety ("ample" being undefined) to setting standards based on maximizing net benefits to society., In some cases, Congress has implicitly directed an agency to perform risk assessment but provided no guidance at all with respect to what to do about it.
THE RISK MANAGEMENT DIRECTIVES OF THE FEDERAL SAFE DRINKING WATER ACT
TCE is a groundwater pollutant, and what Schumer (and the environmentalists he was speaking to) want EPA to do is promulgate a primary drinking water standard lower than 5 ppb. For EPA to do this, the Safe Drinking Water Act has a complicated set of procedures it must follow. These procedures include risk assessment, but there are important statutory criteria for what constitutes an acceptable risk assessment. It must be "objective" and presented in a way that is "comprehensive, informative, and understandable." Moreover, risk assessment is just a starting point in the statutory process. EPA also has to determine that a contaminant is national in scope and that setting a standard would achieve significant public health benefits. (As an aside, nothing in federal law prevents New York from adopting more stringent drinking water standards on its own authority. Orr's article indicates that the state health department is not inclined to do so because it does not believe residents are at risk.)
Some of these statutory procedures are contained in the excerpt below. It establishes very specific provisions for how risk assessment must be performed, and how it must be used if science is going to serve as the foundation for regulatory standards. Schumer presumably is familiar with the requirements of the law; he served in the House of Representatives when these requirements were approved by a vote of 392-30 (though he himself was absent). But none of the three news accounts of his conference call indicate that he communicated this knowledge to reporters. Instead, Schumer appears to have discussed EPA's 2001 draft risk assessment as if it had been a proposed risk management standard.
With this as input, it's unsurprising that two of the three news stories confuse risk assessment and risk management. Relano and Murphy of Newsday follow this confusion to its natural conclusion, making at least one major factual error in each of their first three paragraphs:
The federal government has failed for six years to implement updated standards on TCE, an industrial solvent that has contaminated water in the aquifer under Long Island, Sen. Charles Schumer said Monday.
The Environmental Protection Agency drafted standards in 2001 for TCE, trichloroethylene, "but has left them on the shelf," said Schumer (D-N.Y.). "The EPA is twiddling its thumbs."
Those 2001 draft standards characterized TCE as "highly likely to produce cancer in humans," although there were suggestions that it affected children and adults differently, according to the EPA Web site.
On the distinction between the assessment of risk (a scientific function) and its management (a policy function), Monnat's story is the most confused. He knows the words but is unsure what to do with them:The Environmental Protection Agency drafted standards in 2001 for TCE, trichloroethylene, "but has left them on the shelf," said Schumer (D-N.Y.). "The EPA is twiddling its thumbs."
Those 2001 draft standards characterized TCE as "highly likely to produce cancer in humans," although there were suggestions that it affected children and adults differently, according to the EPA Web site.
In Schumer's latest effort for the EPA to right the situation, he wrote a letter to EPA Administrator Stephen L. Johnson asking for the EPA to strengthen federal TCE requirements and amounts allowed to be found in drinking water, update and finalize the 2001 report including risk management assessment to reflect the carcinogenic nature of TCE and to fully re-evaluate existing sites where TCE is seeping into groundwater and evaporating into the air under the new, tougher standard.
He puts risk management before risk assessment (an impossible sequence), but correctly reported that EPA's 2001 document was a risk assessment and not a standard. Yet he incorrectly thought the purpose of risk assessment was to recommend a risk management decision:[A] TCE Health Risk Assessment was issued by the EPA in 2001 calling for drinking water standards to be toughened, but the EPA has not yet acted on the recommendation.
These errors are common among reporters, often because their sources do not provide accurate information or put it in proper context. Unfortunately, these errors also are common among scientists -- including scientists serving on prestigious NRC committees (more on that below).
Schumer, D-N.Y., said the U.S. Environmental Protection Agency "has turned a blind eye to this growing danger." He called on the agency to toughen regulatory standards and update its health risk assessment of trichloroethene, or TCE, which he said is 20 years out of date.
THE NATIONAL RESEARCH COUNCIL REVIEW
The NRC committee understood its task was to help EPA produce an objective risk assessment but steer clear of risk management because that is the province of EPA officials:
The committee was asked to examine issues critical to developing an objective, realistic, scientifically based health risk assessment for trichloroethylene.
The review itself is largely faithful to this charge, but at several critical junctures the committee confused risk assessment and risk management. In a discussion styled "dose-response assessment," the committee didn't actually do dose-response; instead, it adopted a conventional safety assessment approach that utiizes some of the data and applies risk management-based safety factors to determine a "safe" dose. To escape the inherent risk management content in this, the committee recommended that EPA identify multiple so-called "points of departure" to precede the application of these safety factors. With respect to cancer risk assessment, the committee could not illuminate the dose-response relationship and simply adopted EPA risk management policy guidance. By intent and design, that guidance is not objective. It begins with certain risk management decisions and requires science to meet a high (and unspecified) evidentiary burden that these risk management decisions are wrong. The practical effects of these practices are that they transfer from the EPA Administrator to Agency scientists the authority to exercise the risk management discretion provided to the Administrator by law.
There's no doubt the members of the NRC committee were well-intentioned and sought only to do the best job they could with the information available. However, when even nationally distinguished scientists are not able to discern the boundary between risk assessment and risk management -- or, alternatively, resist the temptation to embed their own opinions about risk management into the assessment of risk -- its no surprise that United States Senators, or the reporters who cover them, do the same thing.
None of this serves EPA well, and ironically it also harms residents of communities in Upstate New York and Long Island who are concerned about TCE in their drinking water. The Agency cannot legally use a purposefully biased risk assessment as its foundation for a primary drinking water standard. Federal law requires that the risk assessment for TCE be objective -- that is, free of embedded risk management considerations.
National Research Council, 2006. Assessing the Human Health Risks of Trichloroethylene: Key Scientific Issues, National Academies Press; press release, report in brief.
(3) Risk assessment, management, and communication.-
(A) Use of science in decisionmaking.--In carrying out this section, and, to the degree that an Agency action is based on science, the Administrator shall use--
(i) the best available, peer-reviewed science and supporting studies conducted in accordance with sound and objective scientific practices; and
(ii) data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies use of the data).
(ii) data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies use of the data).
(B) Public information.--In carrying out this section, the Administrator shall ensure that the presentation of information on public health effects is comprehensive, informative, and understandable. The Administrator shall, in a document made available to the public in support of a regulation promulgated under this section, specify, to the extent practicable--
(i) each population addressed by any estimate of public health effects;
(ii) the expected risk or central estimate of risk for the specific populations;
(iii) each appropriate upper-bound or lower-bound estimate of risk;
(iv) each significant uncertainty identified in the process of the assessment of public health effects and studies that would assist in resolving the uncertainty; and
(v) peer-reviewed studies known to the Administrator that support, are directly relevant to, or fail to support any estimate of public health effects and the methodology used to reconcile inconsistencies in the scientific data.
(iv) each significant uncertainty identified in the process of the assessment of public health effects and studies that would assist in resolving the uncertainty; and
(v) peer-reviewed studies known to the Administrator that support, are directly relevant to, or fail to support any estimate of public health effects and the methodology used to reconcile inconsistencies in the scientific data.
(C) Health risk reduction and cost analysis.--
(i) Maximum contaminant levels.--When proposing any national primary drinking water regulation that includes a maximum contaminant level, the Administrator shall, with respect to a maximum contaminant level that is being considered in accordance with paragraph (4) and each alternative maximum contaminant level that is being considered pursuant to paragraph (5) or (6)(A), publish, seek public comment on, and use for the purposes of paragraphs (4), (5), and (6) an analysis of each of the following:
(I) Quantifiable and nonquantifiable health risk reduction benefits for which there is a factual basis in the rulemaking record to conclude that such benefits are likely to occur as the result of treatment to comply with each level.
(II) Quantifiable and nonquantifiable health risk reduction benefits for which there is a factual basis in the rulemaking record to conclude that such benefits are likely to occur from reductions in co-occurring contaminants that may be attributed solely to compliance with the maximum contaminant level, excluding benefits resulting from compliance with other proposed or promulgated regulations.
(III) Quantifiable and nonquantifiable costs for which there is a factual basis in the rulemaking record to conclude that such costs are likely to occur solely as a result of compliance with the maximum contaminant level, including monitoring, treatment, and other costs and excluding costs resulting from compliance with other proposed or promulgated regulations.
(IV) The incremental costs and benefits associated with each alternative maximum contaminant level considered.
(V) The effects of the contaminant on the general population and on groups within the general population such as infants, children, pregnant women, the elderly, individuals with a history of serious illness, or other subpopulations that are identified as likely to be at greater risk of adverse health effects due to exposure to contaminants in drinking water than the general population.
(VI) Any increased health risk that may occur as the result of compliance, including risks associated with co-occurring contaminants.
(VII) Other relevant factors, including the quality and extent of the information, the uncertainties in the analysis supporting subclauses (I) through (VI), and factors with respect to the degree and nature of the risk.
(IV) The incremental costs and benefits associated with each alternative maximum contaminant level considered.
(V) The effects of the contaminant on the general population and on groups within the general population such as infants, children, pregnant women, the elderly, individuals with a history of serious illness, or other subpopulations that are identified as likely to be at greater risk of adverse health effects due to exposure to contaminants in drinking water than the general population.
(VI) Any increased health risk that may occur as the result of compliance, including risks associated with co-occurring contaminants.
(VII) Other relevant factors, including the quality and extent of the information, the uncertainties in the analysis supporting subclauses (I) through (VI), and factors with respect to the degree and nature of the risk.
(ii) Publication. Treatment techniques.--When proposing a national primary drinking water regulation that includes a treatment technique in accordance with paragraph (7)(A), the Administrator shall publish and seek public comment on an analysis of the health risk reduction benefits and costs likely to be experienced as the result of compliance with the treatment technique and alternative treatment techniques that are being considered, taking into account, as appropriate, the factors described in clause (i).
(iii) Approaches to measure and value benefits.--The Administrator may identify valid approaches for the measurement and valuation of benefits under this subparagraph, including approaches to identify consumer willingness to pay for reductions in health risks from drinking water contaminants.
(iv) Authorization.--There are authorized to appropriated to the Administrator, acting through the Office of Ground Water and Drinking Water, to conduct studies, assessments, and analyses in support of regulations or the development of methods, $35,000,000 for each of fiscal years 1996 through 2003.
(iv) Authorization.--There are authorized to appropriated to the Administrator, acting through the Office of Ground Water and Drinking Water, to conduct studies, assessments, and analyses in support of regulations or the development of methods, $35,000,000 for each of fiscal years 1996 through 2003.
