15 Apr 2009
(A)H1N1 Influenza:
Science and policy collide on defining a 'pandemic'
by Richard Belzer
in Information Quality, Regulatory Policy, Regulatory Science
On June 11 the World Health Organization decreed that the recent outbreak of (A)H1N1 influenza ("swine flue") qualified as a "pandemic." According to reporters for the Washington Post, WHO delayed making this decision long after it technically met its established definition. The reason for delay is that policy officials no longer liked the definition and were concerned that a declaration of a pandemic could lead to panic.
More...30 Mar 2009
Forecasting Economic Depression:
Illustrating the pitfalls of expert elicitation
by Richard Belzer
in Information Quality, Regulatory Science
In recent years there has been a notable increase in the use of expert elicitation in human health risk assessment. The method usually involves empaneling a group of experts and, through a carefully crafted and complex set of procedures, asking each panel member to provide a subjective probability that some phenomenon that cannot be directly observed is true or false. The Environmental Protection Agency has an informative external review draft white paper on the subject.
In environmental health, expert elicitation has been used to quantify the risk of cancer from drinking water disinfection byproducts, the likelihood that routine exposure to particulate matter in air causes premature mortality, and the magnitude of uncertainties related to climate change. Each is a tough scientific question. For example, the risk involved may be quantitatively small, and hence hard to discern, or the scientific uncertainties may be very large. Judgment is required, and the judgments of scientists inevitably reflect a mixture of scientific expertise and nonscientific opinion.
The need for discerning science from policy in expert judgment has been recognized for decades, at least since the 1983 National Research Council Red Book. No consensus yet exists concerning how to do this in practice. As a result, practitioners of expert elicitation typically acknowledge the problem but not much else. The EPA external review draft white paper mentioned above, for example, says that Agency technical support documents relying on expert elicitation should address "[p]ossible correlations with non-elicited components of the overall analysis or policy question" -- a phrase that, when translated into plain English, means the inflitration of experts' policy views into their characterization of science.
Today's Wall Street Journal has an example drawn from a very different arena -- macroeconomic forecasting -- that offers a wealth of insight about the problems with expert elicitation.
More...15 Mar 2009
Gray Wolves:
When does science end and policymaking begin?
by Richard Belzer
in Information Quality, Regulatory Policy, Regulatory Science
Washington Post environmental reporter Juliet Eilperin has a Page One story about Department of the Interior Secretary Ken Salazar's decision to ratify the Bush Administration's decision in December 2008 to delist the gray wolf as an endangered species in parts of the West.
His decision is controversial. Those who support it (including of course Secretary Salazar) say it was based on science. Those who oppose it say the Bush Administration's decision was based on politics and that Salazar should have changed it to reflect the policy views of the Obama Administration. One thing is clear: it is difficult to discern where science ends and policy begins with respect to decisions made under the Endangered Species Act.
More...6 Mar 2009
Losing Health Care Due to Bankruptcy:
Is there one case every 30 seconds?
by Richard Belzer
in Information Quality, Regulatory Economics, Regulatory Policy
President Obama hosted a "health care summit" yesterday, during which he made an astounding claim, according to Dan Lothian, White House Correspondent for CNN:
"The cost of health care now causes a bankruptcy in America every 30 seconds. By the end of the year, it could cause 1.5 million Americans to lose their homes," according to an excerpt of his opening remarks released by the White House.
Could this be true?
More...4 Feb 2009
Economics in Theory and Practice:
The American Economic Association surveys its members, badly
by Richard Belzer
in Amusements, Information Quality
Attendees of the annual American Economics Association annual meeting received an email asking them to participate in a survey:
Every five or six years AEA does a survey of the economists who attend the ASSA meetings. The survey results affect decisions concerning future annual meetings. Please help us out by taking a few minutes to complete the survey.
This "survey" produces information with little or no value.
More...21 Dec 2008
Obama Science Nominees:
OSTP and NOAA
by Richard Belzer
in Information Quality, Regulatory Policy, Regulatory Science
President-elect Obama has announced that he will make the following nominations to important science-related offices:
John Holdren
Assistant to the President for Science and Technology
Director, Office of Science and Technology Policy
Jane Lubchenco
Administrator, Nationol Oceanic and Atmospheric Administration (NOAA)
Department of Commerce
12 Nov 2008
Petition for Correction to the Office of Management and Budget Concerning the Draft 2008 Report to Congress on the Benefits and Costs of Federal Regulation
by Richard Belzer
in Information Quality
On November 7, 2008, Regulatory Checkbook -- Neutral Sourcve's sister nonprofit organization -- filed a public comment on the Office of Management and Budget's draft 2008 Report to Congress on the Benefits and Costs of Federal Regulation.
This public comment also was submitted as a formal Petition for Correction under OMB's Information Quality Guidelines (PDF). Under those guidelines, influential information OMB disseminates must be substantively and presentionally objective, transparent and reproducible, and provide utility for its intended purposes of informing Congress and the public.
The draft Report does not satisfy these information quality standards.
OMB is obligated by its own guidelines to respond within 60 days (i.e., by January 6, 2009).
More...
30 Jun 2008
Rounding Error and Information Quality:
The case of Exxon Shipping Co. v. Baker
by Richard Belzer
in Information Quality
Last week the Supreme Court reversed an appellate court opinion that would have imposed $2.5 billion in punitive damages resulting from the 1989 Exxon Valdez oil spill. In the majority opinion written by Justice David Souter, the Court opined on a matter of maritime law for which there was neither a constitutional precedent nor operating law. The Court ruled that a 1:1 ratio of punitive to compensatory damages "is a fair upper limit in such maritime cases."
The Court obtained this ratio by conducting an ad hoc qualitative statistical analysis of trial court practice, which yielded a ratio of 0.65:1, then rounding up to 1:1. It is instructive to note the practical financial significance of the arcane information quality issue of significant figures (see here and here). More...
29 May 2008
Information Quality and Peer Review:
Are disclaimers in draft documents effective?
by Richard Belzer
in Information Quality
Since 2002, federal information quality guidelines have required agencies to avoid disseminating scientific information that is not objective, and to have effective administrative systems for managing requests for the correction of information that a petitioner believes is incorrect. The burden of proof of error rests with the petitioner.
All information that is "disseminated" is covered by these rules, but information that is made public solely for the purpose of scientific peer review or public comment is exempt from the definition -- provided that it is accompanied with a specified disclaimer (p. 8):
The purpose of this disclaimer is to deter people from relying on draft documents. An empirical question is whether the prescribed language is strong enough.
An interesting test case has arisen with respect to the industrial chemical bisphenol A (BPA). More...
1 Apr 2008
Regulating How Drop-Out Rates Are Reported:
The tip of the iceberg of a persistent information quality problem
by Richard Belzer
in Information Quality
Department of Education secretary Margaret Spellings has announced a new regulation to control how states report drop-out rates. Under existing law, the states have the discretion to devise their own formulas. This makes interstate comparisons problematic. It also reflects the states' interest in devising formulae that under-report actual drop-out rates.
Under the proposed rule, all states would have to use the same federally prescribed formula. More...
3 Jan 2008
Where to Have a Cardiac Arrest?
Answer: not in the hospital
by Richard Belzer
in Information Quality, People & Institutions, Regulatory Science
New York Times reporter Denise Grady previews a research report due to be published in today's New England Journal of Medicine that says many hospitals do not respond quickly enough to cardiac arrest. Leslie Saxon, who wrote an accompanying editorial, delivered the money quote: “You’re better off having your arrest at Nordstrom, where I’m standing right now, because there are 15 people around me.” More...
6 Dec 2007
Objectivity in Risk Assessment:
The National Intelligence Estimate on Iran, Part 3
by Richard Belzer
in Information Quality, Regulatory Policy
The web is chock full of commentary on the recently released summary of the National Intelligence Estimate on Iran. (We emphasize summary because the body of the NIE remains classified.) We've posted here and here on the NIE as a risk assessment document, noting that it claims to be an objective assessment not confounded by risk management (i.e., policy or political) concerns.
We've read much (but by no means all) of this news and commnetary and drawn some inferences we hope are useful. More...
19 Oct 2007
DHS' "No-Match Rule" Stopped by Preliminary Injunction:
The Regulatory Flexibility Act and illegal aliens
by Richard Belzer
in Information Quality, Litigation, Regulatory Policy
On October 10, U.S. District Judge Charles Breyer issued a preliminary injunction barring the Department of Homeland Security from implementing a regulation it issued in August that tightened up existing practice in the enforcement of 1986 federal immigration law. The case provides a lesson in administrative procedure -- particularly, how an agency's failure to take obscure procedures seriously can backfire. More...
8 Jul 2007
Ranking Public School Quality (Badly), Part 2:
Does per pupil spending predict average SAT scores?
by Richard Belzer
in Information Quality
Following up on yesterday's post, in which we pointed out numerous information quality errors in the high school ranking by Christina Settimi and published by Forbes, we decided to address this question using her data. More...
7 Jul 2007
Ranking Public School Quality (Badly):
Forbes' has problems with information quality
by Richard Belzer
in Information Quality
School quality is something about which everyone seems to have an opinion, and a compulsion to rank. For example, every year there is a kerfuffle about the college rankings published by US News and World Report. This year the ranks of college administrators refusing to provide data to US News has grown because opponents have become more organized.
Elementary and secondary school rankings are the latest trend, and the number of rankings can be expected to grow as more statistics are made available. What do these rankings actually mean?
More...
