Login

Home > Blog

"Disentangling Bias from Central Tendency Estimates"
Fixing OMB's proposed risk assessment guidance

19 Dec 2006 in Regulatory Science, Regulatory Policy, Information Quality

On December 6 Neutral Source managing editor Richard Belzer delivered a presentation at the annual meeting of the Society of Risk Analysis on certain aspects of OMB's proposed guidance on risk assessment.

In response to numerous requests for copies, the presentation (with notes) is posted here.

This presentation was delivered at the 2006 Society for Risk Analysis Annual Meeting, December 6, 2006, in Baltimore MD.

© Richard B. Belzer 2006. All rights reserved.

Society for Risk Analyis
2006 Annual Meeting
Baltimore, MD

Session W1-E:
Economists responses to the OMB risk assessment bulletin

The abstract is here.

---

HISTORY

• 1995: Paperwork Reduction Act Amendments of 1995 authorized OMB to issue guidelines on information quality. OMB did not exercise this authority.
• 2000: Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554) directed OMB to issue government-wide guidelines (44 U.S.C. 3504(d)(1)) or rules (44 U.S.C. 3516) by September 30, 2001.
  • OMB was to “provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies in fulfillment of the purposes and provisions of chapter 35 of title 44, United States Code" (Paperwork Reduction Act).
  • Congress did not define “quality,” “objectivity,” “utility,” or “integrity.” Thus, it delegated to OMB the legislative authority to define these terms.
• 2001: OMB issued interim final government-wide information quality guidelines.
• 2002: OMB issued final government-wide information quality guidelines (the "IQG").
  • OMB defined “objectivity” in terms of both “presentational” and “substantive” objectivity.
  • Coverage:
  • All agencies covered by the Paperwork Reduction Act are covered by IQG.
  • Covered agencies were directed to issue their own agency-specific guidelines on or before October 1, 2002. Almost all agencies met that deadline.
  • Agency-specific guidelines may add but cannot subtract from OMB's IQG. In any conflict between OMBs IQG and an agency-specific guideline, OMB's IQG applies.
• 2006: OMB proposes risk assessment guidance.
  • The guidance is not new, but an application of the IQG to human health, safety and environmental risk assessment.
  • They include text related to “objectivity,” “unbiasedness,” and “central tendency.”

The premise of this paper is that the text (and especially the preamble) of the proposed risk assessment guidance confuses these terms, but that with relative simple changes this confusion can be substantially eliminated.

OMB’S RESPONSIBILITIES ARE BROADER THAN JUST INFORMATION QUALITY

• Centralized regulatory review is the best-known authority, but not the only authority relevant to the proposed risk assessment guidance.
  • 1981-93: Executive order 12291.
  • 1993-date: Executive order 12866.
  • Both implicitly require objective analysis of benefits and costs.
  • Though Circular A-4 is the current incarnation, OMB has had guidance on regulatory analysis since at least 1990 (published in the annual “Regulatory Program of the United States Government” pursuant to Executive order 12498)
  • OMB regulatory analysis guidance has always called for objectivity in estimation.
• The Paperwork Reduction Act is the foundation of the IQG (already discussed), though knowledge and understanding of the PRA is limited.
• Since 1998, OMB has been required to report annually to Congress on the benefits and costs of federal regulation. This requirement, called the Regulatory Right-to-Know Act (”RRTK”), is codified at 31 U.S.C. 1105 note.
• None of OMB's reports to Congress has adhered to the OMB IQG because none of them reports objective estimates.
• OMB is therefore highly susceptible to an error correction petition claiming that it’s own information disseminations are contrary to law.
• OMB lacks the capacity to generate its own estimates except for a handful of cases, and it is institutionally predisposed to keep these estimates confidential. Thus, OMB is dependent either on more objective estimates from agencies or estimates from third parties.
  • The IQG establishes principles that third parties can use to produce superior estimates.
  • The proposed risk assessment guidance can be justified as an effort to promote both.

WHAT DOES “OBJECTIVITY” MEAN?

• “Presentational” objectivity and “substantive” objectivity are defined in similar ways using overlapping language that itself is left undefined. Hence, OMB’s definitions require careful analysis to understand. (That these definitions require careful analysis to understand is sufficient evidence that they are problematic.)
• Ah, the scientific method can rescue us!
  • Agencies can be granted a default presumption of objectivity.
  • With that as the “null” hypothesis, affected parties (and OMB itself) can refute this weak presumption by documenting evidence that information is not accurate, clear, complete or unbiased (to disprove the default of “presentational” objectivity) or that it is not accurate, reliable or unbiased (to disprove the default of “substantive” objectivity).
  • What burden of proof must be presented to refute this default assumption of objectivity?
    • Something akin to the classical statistical test (p<0.05) is close to a “beyond reasonable doubt” standard. If applied, it would make it very difficult for third parties or OMB to refute the default. A vast amount of information that does not adhere to the objectivity standards would be permitted.
    • A“preponderance of the evidence” test would permit fewer violations of the objectivity test, but still would permit many. Also, it’s unclear just what probability of falsehood is implied by this test. A “more likely than not” test is a special, limiting case of the preponderance of evidence test.
    • A “significant evidence” test would permit even fewer violations, and depending on how low the bar is set for significance, it could effectively place the burden of proof on an agency once it had been challenged.
• OMB’s guidance is ambiguous on the evidentiary burden. A default presumption of objectivity can be refuted by “persuasive” evidence.
  • The guidance does not specify how persuasive, nor to whom.
  • Like in litigation, the proper evidentiary standard may depend on the magnitude of Type I and Type II errors, and the costs associated with suffering them.
• Refutation by scientific method is strongly preferred to the conventional “reasoned determination” test that agencies typically must satisfy to justify decisions. Reasoned determinations may make sense for policy decisions, but they are ill-suited for determining the soundness of underlying facts. If objectivity is decided by “reasoned determination,” skilled argumentation will rule over science.
  

OBJECTIVITY AND BIAS

• I’ve listed four types of bias that could be captured by OMB’s use of the term. Indeed, a serious defect of OMB’s definition is that it appears to use “bias” in multiple ways without making clear which meaning applies in which context.

STATISTICAL BIAS

• Statistical bias happens. In virtually every application of statistical methods we know, statisticians try hard to avoid bias and have developed fairly complex tools for discerning it and removing it.
• Except in the context of human health risk assessment, where biostatisticians develop explicitly biased estimators. The linearized multistage model and benchmark dose are classic examples.

POLICY-DRIVEN BIAS

• The last three types of bias share common denominator – they capture the nonscientific judgments of the risk assessor.
• In decision-making, any of these three types of bias may be perfectly fine. But these situations all involve risk management, not risk assessment.
  • None of these three types of bias is necessarily wrong in risk management
  • They are always wrong in risk assessment
• This is what the proposed risk assessment guidance says, and which its antecedent Information Quality Guidelines already implied: it is inappropriate to include any of these policy-based judgments in risk assessment.
• It is impossible for a risk assessment that contains these policy-driven biases to adhere to the objectivity standard in OMB’s 2002 Information Quality Guidelines.
  • This is not just OMB’s doing
  • It is impossible for risk assessments containing such biases to comply with the law that directed OMB to issue these guidelines unless "objectivity" is defined as its antonym.

FULL DISCLOSURE IS NOT A SOLUTION

• Some agencies have adopted the practice of disclosing default assumptions, embedded risk management preferences, and precautionary judgments or values that are contained in risk assessment. This is a welcome reform, as anything that makes risk assessment more transparent is beneficial.
• However, these agencies’ motivating hope appears to be that as long as they are transparent, these forms of bias are acceptable risk assessment practices.
• They are not. Transparency is not a substitute for objectivity.
  • If a judge were to honestly list all the reasons why a defendant’s trial is rigged against him, would that substitute for a fair trial?
  • If a police officer were to transparently reveal to a motorist that his radar detector is set to add an average of 20 mph to every reading, would that mean a reading of 80 mph is always a violation of a 65 mph speed limit?
  • If any agency were to list all the reasons why its estimate of risk is unreliable or invalid, would that make the resulting risk estimate reliable or valid?
  • If an agency were to clearly list all the reasons why its risk assessment does not adhere to applicable information quality guidelines, would that be evidence of adherence to the guidelines?
• Transparency is a necessary but not sufficient procedural test for objectivity.
  • Transparency is essential to enable the agency’s default presumption of objectivity to be tested by third parties and OMB staff.
  • Without transparency, an agency should lose even the weakest presumption of objectivity.
  • If the author of a risk assessment declines to make it transparent, it is fully appropriate to infer that the risk assessment does not adhere to applicable information quality guidelines.
• Federal agencies face a simple dilemma:
  • Risk assessments are covered by the IQG if they are scientific determinations and exempt of they are policy judgments.
    • If they are scientific determinations, then they must adhere to the IQG standard of objectivity.
    • If they are policy judgments, then they are exempt from the IQG but covered by applicable provisions of the Administrative Procedure Act.

TWO EXAMPLES FROM OUTSIDE HUMAN HEALTH RISK ASSESSMENT

1. S&L MELTDOWN

• In the early 1990s, a “meltdown” of sorts occurred in the savings & loan industry. Many S&Ls went bankrupt.
• Congress and the Bush 41 administration collaborated on a rescue initiative. The Resolution Trust Corporation was created to liquidate assets.
• The question government officials asked was this: “What is the federal government’s financial exposure under various intervention scenarios?”
  • They wanted unbiased estimates of USG financial exposure.
  • They had every intention of making risk-averse decisions, but they wanted facts, not opinions, from civil service analysts.

2. SOCIAL SECURITY TRUST FUND

• The Trustees report annually on the financial solvency of the trust fund, including a projection of the date by which revenue from taxes will fall below outlays.
  • The latest estimate is 2040.
  • There is uncertainty; the predicted year varies +/- several years under the most plausible scenario, and it differs considerably by scenario.
• Everyone agrees that the date should be estimated without embedded policy preferences concerning what should be done about it.
• The fastest way to be disenfranchised from the debate is to purposefully jigger the numbers.

In finance, statistical bias and embedded policy judgment are impermissible in risk assessment. In human health risk assessment, they are the norm.

A FIGURE FROM THE LATEST TRUSTEE REPORT

• Shows predicted tax revenue less outlays, 2005 -- 2080.
• The estimate is intended to be unbiased – both in the statistical sense and in the colloquial sense of not having policy preferences embedded in it.
• The estimate does not show uncertainty -- but it could, and probably should, under proposed risk assessment guidance.
• Why is it justifiable for human health risk assessments to include such biases when estimates like these do not?

WHY OMB NEEDS CENTRAL TENDENCY ESTIMATES

• OMB appears to mean “unbiased” when it refers to “central tendency” risk estimates. That’s because central tendency risk estimates are needed for valid estimates of the benefits of regulatory intervention. But the language is not clear.
• OMB’s reports to Congress on the benefits and costs of federal regulation have been justly criticized as invalid and unreliable. See, e.g., http://neutralsource.org/content/article/detail/685 for my 2004 testimony before the House of Representatives.
  • One reason is the propensity of federal agencies to develop upper-bound estimates of risk, then use these upper-bound estimates in benefits assessment as if they represented the mean.
  • Objective reporting by OMB requires objective estimates of:
    • The amount of baseline risk
    • The amount of risk reduction expected after regulatory intervention
    • The amount of countervailing risk expected as a result of the regulatory intervention (whether direct, via substitution effect, or via income effect)
    • The value to those whose risks would be reduced of the amount of risk expected to be reduced. Biased estimates of valuation are violations of the IQG, just as are biased risk assessments, Bias occurs several ways:
      • Failing to properly discount. (The appropriate discount rate is the rate of time preference of the subpopulation that would realize risk reduction.)
      • Failing to properly account for lag effects in the realization of risk reduction. (Long-standing practice is to assume risk reductions are realized immediately after exposure is reduced or ceases.)
      • Using a VSL when the number of life-years is significantly greater or less than about 35 (the age for which VSL values apply)
      • Using a VSL or VSLY that applies to different subpopulation than the one in which risk reductions would be realized.
  • Cost (conventionally assumed to be overestimated)
    • Cost is biased upward to the extent that a regulation is based on performance standards and estimates do not account for innovation. But it is biased downward if the assumptions about innovation are unreasonably charitable.
    • Cost is biased downward to the extent that transactions costs are ignored.
    • Cost is biased downward to the extent that the cost of transition is ignored.
    • Cost is biased downward to the extent that it is approximated by expenditure. This may be the greatest source of bias because:
      • A lot of cost does not show up as expenditure.
      • Cost is properly understood as “opportunity cost.”
      • “Opportunity cost” means benefits foregone. Agencies rarely or never estimate benefits foregone.

WHAT TO DO?

1. LIMIT THE USE OF “BIAS” TO ITS STATISTICAL MEANING.
2. REPLACE “BIAS” WITH “ABSENCE OF POLICY NEUTRALITY” FOR ALL OTHER PURPOSES

• Because of its multiple meanings, the term “bias” with respect to science and risk assessment should be restricted to its statistical meaning, where it has a clear definition and no synonyms.
• The other examples of “bias” listed are policy-driven.
  • That they are policy-driven does not make them wrong, but it is wrong to use them in risk assessment because they are the province of risk management.
  • When they appear in risk assessment, they usurp authorities delegated by Congress to policy officials.
    • Sometimes, this bothers policy officials greatly and sometimes it provides a welcome way to evade accountability (“the scientists made me do it”).
    • Policy officials who insist on exercising their delegated authorities in ways contrary to the policy preferences embedded in risk assessment are unfairly accused of “interfering with science.”
• The essence of “objectivity” is the absence of favoritism, however it might be expressed.
  • Some of the language in OMB’s proposed text can be adapted easily to fit this approach – e.g., risk assessments must “neither understate nor overstate” risk.
  • To be more explicit and clear, however, it is vital that risk assessment “not materially favor or disfavor any particular or genre of policy alternatives that might be devised for risk management." The adverb “materially” is intended to exclude trivial matters.
• Returning to our previous point about the evidentiary burden, only a weak presumption of objectivity is appropriate. Significant evidence of under- or overstating risk, or materially favoring or disfavoring a policy position, should be persuasive evidence of the absence of objectivity.

WHY CENTRAL TENDENCY ESTIMATES

• There are at least three powerful reasons why risk assessments must include central tendency estimates to be objective
  1. Presentational objectivity under the IQG requires it.
  2. They are needed for benefit-cost analysis, which for major federal regulations has been required for 25 years.
  3. OMB’s annual reports to Congress under the Regulatory Right to Know Act cannot satisfy OMB’s own objectivity test if benefit and cost estimates are not based on central tendency estimates.
• Many other points on a risk distribution are important, and they may well be needed for risk management decision making.
• Regardless of which percentile of the distribution is of interest for risk management purposes, it should be estimated using unbiased methods.

TAKE HOME MESSAGES

1. OMB has multiple interests served by the proposed risk assessment guidance. Among them (but thus far neglected) is OMB’s need for objective information about risk to fulfill its obligations under the Regulatory Right to Know Act (RRTK).
   1. OMB’s reports to Congress since 1998 have all been deeply flawed because they do not report objective estimates of benefits and costs.
   2. These reports violate OMB’s own information quality guidelines.
   3. To fix these problems, OMB needs more objective information on which to base its reports.
2. OMB’s definition of “objectivity” is helpful because it is refutable, thus permitting the application of scientific method. OMB is unduly ambiguous, however, about the evidentiary burden that must be met to successfully refute this default.
3. Bias is a useful construct but it has too many meanings, some of which are frankly pejorative.
   1. It is better to limit the term “bias” to its statistical meaning and use different language to describe other problems, such as embedded policy preferences.
   2. The term “policy neutral” captures the essence of what OMB means by the absence of bias when it uses the term in a non-statistical context. We should use that language instead.
4. Examples from outside human health risk assessment show that it is normal and expected that risk assessments be unbiased, or free of embedded policy preferences. There is no justification for applying a different and lower standard on human health risk assessment.
5. Central tendency estimates are essential for objective reporting to Congress on the benefits and costs of federal regulation. That goes for valuations of risk reduction and cost as well as risk assessment.

Questions or comments? Contact the author:

Richard B. Belzer PhD
President, Regulatory Checkbook
Managing Editor, Neutral Source

[add a comment]

Add a Comment

Contact | © 2008 Neutral Source | About