30 Jun 2008
Rounding Error and Information Quality:
The case of Exxon Shipping Co. v. Baker
by Richard Belzer
in Information Quality
Last week the Supreme Court reversed an appellate court opinion that would have imposed $2.5 billion in punitive damages resulting from the 1989 Exxon Valdez oil spill. In the majority opinion written by Justice David Souter, the Court opined on a matter of maritime law for which there was neither a constitutional precedent nor operating law. The Court ruled that a 1:1 ratio of punitive to compensatory damages "is a fair upper limit in such maritime cases."
The Court obtained this ratio by conducting an ad hoc qualitative statistical analysis of trial court practice, which yielded a ratio of 0.65:1, then rounding up to 1:1. It is instructive to note the practical financial significance of the arcane information quality issue of significant figures (see here and here). More...
29 May 2008
Information Quality and Peer Review:
Are disclaimers in draft documents effective?
by Richard Belzer
in Information Quality
Since 2002, federal information quality guidelines have required agencies to avoid disseminating scientific information that is not objective, and to have effective administrative systems for managing requests for the correction of information that a petitioner believes is incorrect. The burden of proof of error rests with the petitioner.
All information that is "disseminated" is covered by these rules, but information that is made public solely for the purpose of scientific peer review or public comment is exempt from the definition -- provided that it is accompanied with a specified disclaimer (p. 8):
The purpose of this disclaimer is to deter people from relying on draft documents. An empirical question is whether the prescribed language is strong enough.
An interesting test case has arisen with respect to the industrial chemical bisphenol A (BPA). More...
1 Apr 2008
Regulating How Drop-Out Rates Are Reported:
The tip of the iceberg of a persistent information quality problem
by Richard Belzer
in Information Quality
Department of Education secretary Margaret Spellings has announced a new regulation to control how states report drop-out rates. Under existing law, the states have the discretion to devise their own formulas. This makes interstate comparisons problematic. It also reflects the states' interest in devising formulae that under-report actual drop-out rates.
Under the proposed rule, all states would have to use the same federally prescribed formula. More...
3 Jan 2008
Where to Have a Cardiac Arrest?
Answer: not in the hospital
by Richard Belzer
in Information Quality, People & Institutions, Regulatory Science
New York Times reporter Denise Grady previews a research report due to be published in today's New England Journal of Medicine that says many hospitals do not respond quickly enough to cardiac arrest. Leslie Saxon, who wrote an accompanying editorial, delivered the money quote: “You’re better off having your arrest at Nordstrom, where I’m standing right now, because there are 15 people around me.” More...
6 Dec 2007
Objectivity in Risk Assessment:
The National Intelligence Estimate on Iran, Part 3
by Richard Belzer
in Information Quality, Regulatory Policy
The web is chock full of commentary on the recently released summary of the National Intelligence Estimate on Iran. (We emphasize summary because the body of the NIE remains classified.) We've posted here and here on the NIE as a risk assessment document, noting that it claims to be an objective assessment not confounded by risk management (i.e., policy or political) concerns.
We've read much (but by no means all) of this news and commnetary and drawn some inferences we hope are useful. More...
19 Oct 2007
DHS' "No-Match Rule" Stopped by Preliminary Injunction:
The Regulatory Flexibility Act and illegal aliens
by Richard Belzer
in Information Quality, Litigation, Regulatory Policy
On October 10, U.S. District Judge Charles Breyer issued a preliminary injunction barring the Department of Homeland Security from implementing a regulation it issued in August that tightened up existing practice in the enforcement of 1986 federal immigration law. The case provides a lesson in administrative procedure -- particularly, how an agency's failure to take obscure procedures seriously can backfire. More...
8 Jul 2007
Ranking Public School Quality (Badly), Part 2:
Does per pupil spending predict average SAT scores?
by Richard Belzer
in Information Quality
Following up on yesterday's post, in which we pointed out numerous information quality errors in the high school ranking by Christina Settimi and published by Forbes, we decided to address this question using her data. More...
7 Jul 2007
Ranking Public School Quality (Badly):
Forbes' has problems with information quality
by Richard Belzer
in Information Quality
School quality is something about which everyone seems to have an opinion, and a compulsion to rank. For example, every year there is a kerfuffle about the college rankings published by US News and World Report. This year the ranks of college administrators refusing to provide data to US News has grown because opponents have become more organized.
Elementary and secondary school rankings are the latest trend, and the number of rankings can be expected to grow as more statistics are made available. What do these rankings actually mean?
More...
5 Jul 2007
Getting Better Information About Air Travel Delays:
How benefit-cost analysis and information quality principles can solve the problem
by Richard Belzer
in Information Quality
In the wake of JetBlue's latest problems, air travel in general is getting more attention. Much of this is focused on various proposals for air travel "customers bills of rights." We've shown that CBORs don't (and possibly can't) address every circumstance of interest. They also have the peculiar effect of taking service quality out of the domain in which airlines can compete. An alternative approach not only allows airlines to compete on service quality, but encourages them to do so.
One requirement for this to work is passengers have to have access to high-quality information that addresses the kind of service quality questions they care about. The Department of Transportation has issued regulations concerning air travel delays. There is mounting evidence that these rules have not yielded the kind of information passengers need.
More...29 Jun 2007
Distinguishing Risk Assessment from Risk Management:
Telling them apart can be hard
by Richard Belzer
in Information Quality, Peer Review, Regulatory Science
Experts in risk analysis often distinguish between risk assessment and risk management. But sometimes they don't, and that can leave the public confused about the difference.
Today's example is trichloroethylene, commonly called TCE. More...
29 May 2007
Journal Peer Review and Objectivity:
The case of Avandia (rosiglitazone) and the New England Journal of Medicine
by Richard Belzer
in Information Quality, Regulatory Science
We've been a consistent supporter of high standards for information quality, especially for federal agencies that disseminate influential information in the support of regulatory objectives. We've also been consistently concerned about one particular aspect of the information quality and peer review guidelines issued by the Office of Management and Budget: the rebuttable presumption that scientific, technical, economic or statistical information published in peer reviewed journals satisfies the guidelines' standard for objectivity.
In the past week, the New England Journal of Medicine published online an article by Steven Nissen and Kathy Wolski claiming that Avandia, GlaxoSmithKilne's blockbuster drug for type 2 diabetes mellitus, "is associated with a significant increase in the risk of myocardial infarction [heart attack] and with an increase in the risk of death from cardiovascular causes that had borderline significance."
The Nissen & Wolski study has received enormous press attention. In today's Wall Street Journal former Food and Drug deputy commissioner Scott Gottlieb asserts that the NEJM has political motives for subjecting the study to inadequate peer review and publishing the study with excessive fanfare. Though not directed at federal information quality law and policy, Gottlieb's commentary suggests circumstances when OMB's rebuttable presumption of objectivity ought to be replaced with a rebuttable presumption of bias. More...
14 May 2007
Socially Motivated Mutual Funds:
Are costs and benefits transparent to investors?
by Richard Belzer
in Information Quality, Regulatory Economics
Jeff Brown in the Washington Post business section discusses "socially responsible" investing, the practice of restricting the domain of one's portfolio to support various causes. Brown provides estimates of how much return on investment soc ially responsible investors sacrifice to pursue their social goals.
Brown's article is remarkable because it is one of few to acknowledge that the imposition of political or social constraints generally leads to lower returns. Investors with strong political or social values can choose to accept lower returns. The problem is mutual funds that promote social causes often imply in their advertising that their returns are just as good as mutual funds that do not impose such restrictions. In other words, they imply there is either no opportunity cost to "socially responsible" investing, or the opportunity cost is minor.
Brown provides some data that put these opportunity costs in perspective. His data, which he attributes to Morningstar, suggest that the opportunity costs of socially motivated investing can be very large. And it's not clear what benefits investors actually get in the process. More...
11 May 2007
Government-wide Information Quality Guidelines:
Does journal peer review achieve "adequate" objectivity?
by Richard Belzer
in Glossary, Information Quality, Peer Review
Federal guidelines require information disseminated by federal agencies to satisfy a few broad criteria, one of which is objectivity. These guidelines give a "rebuttable presumption" to scientific information published in scholarly journals.
- Do all scholarly journals transmit this rebuttable presumption of objectivity? What about scholarly journals that also have an advocacy mission?
- What is the burden of proof for mounting a successful rebuttal?
More...
29 Apr 2007
Estimating Greenhouse Gas Emissions:
The Washington Post and information quality
by Richard Belzer
in Information Quality
A Page One story by Washington Post staff writer David A. Fahrenthold says carbon dioxide emissions in the Washington, DC, area increased 13.4% from 2001 to 2005.
The article clearly is intended to influence public policy. But there are significant problems with this estimate that are not disclosed in the article. The federal Information Quality Act does not apply to the Washington Post, but it would apply to any federal agency that attempted to either take action based on them, or even to report them in a manner suggesting that it thought they were valid. (Congress is exempt from the statutory requirement to only disseminate scientific and statistical data that meet applicable information quality standards. Unlike Executive branch agencies, of course, Congress is never regarded as an authoritative body for scientific or statistical information. )
Below we compare the data reported by Fahrenthold with the information quality standards that apply to federal agencies. More...
9 Mar 2007
The Atlanta Bus Crash
Useful information from a risk analytic perspective
by Richard Belzer
in Information Quality, Regulatory Science
The tragic bus crash outside Atlanta killed five Bluffton University baseball players, the driver, and his wife. News reports indicate that many more remain hospitalized, some in serious or critical condition.
There is a great deal of commentary in the press and in the blogosphere, much of it focused on assigning blame. We are not going to add to that.
We provide links to the most useful information we've found of interest to highway risk analysts, who take seriously their commitment to improve safety but also realize that they cannot prevent all catastrophic bus crashes. More...
