Objectivity in Risk Assessment:
The National Intelligence Estimate on Iran, Part 2
4 Dec 2007 in Regulatory Policy
Early today we posted the unclassified summary of the recent National Intelligence Estimate on Iran's nuclear intentions and capabilities. We said that while we could not comment on its substance -- the details leading to the published judgments remain classified -- the document itself appeared to be notable for its transparency of language, recognition of uncertainty, and the clarity with which it defined the probabilistic language used to describe both the substantive judgments reported and analysts' confidence in them.
Today's news reporting on the NIE shows why it is so important that risk assessments be performed in an objective manner: risk managers, other political actors, and the press often want risk assessments to support predefined narratives and policy views. In less than a day, many of them have stripped away the NIE's careful calibration of uncertainty, suggested that its judgments are colored by the expectation that it would be made public, and undermined the case for performing risk assessment by using it for partisan political purposes.
The Wall Street Journal account published on December 3 (no longer available on the Journal's web site) was a factual reporting of the NIE and provided a link to the text. However, today's reporting, by the Journal (Annelina Lobb; Jay Solomon and Sioban Gorman) and its competitors (for example, Mark Mazetti and Steven Lee Myers, New York Times; Dafna Linzer and Joby Warrick, Washington Post; Richard Wolf and Richard Willing, USA Today; Gregor Peter Schmitz, Spiegel Online), consists of reporters' personal narratives salted with supporting quotes from various political actors, virtually all of whom have strongly held prior views,
This leads both the reporters and those whom they choose to quote to ignore or discount the uncertainties in the NIE. Further, they all miss two crucial risk assessment lessons one can infer from the unclassified summary:
- The change in judgment is the result of new information recently obtained; it is claimed not to be a recalibration of judgment based on the same body of information that was available when the NIE on Iran was last updated in 2005.
- The Intelligence Community widely believes that this new information (whatever it is) is highly credible -- so credible, in fact, that it overwhelms the IC's prior consensus judgment based on all other information in the intelligence database.
The second of these inferences is unambiguously good news. It is a fundamental principle of risk analysis that it is always a good thing when new, high-quality information reduces uncertainty. The reduction of uncertainty is the best reason for performing research in the first place; if research fails to reduce uncertainty, then it has little or no value for decision-making.
Of course, both of these inferences could be disproved, though not without, at a minimum, having access to the entire classified report. For example, it is possible that the new information is not as powerful as the NIE suggests and that the existence of this new information is being used to change prior judgment without admitting so. Similarly, it is possible that the group dynamics within the IC have changed since 2005. This could happen simply due to changes in personnel, because different analysts looking at exactly the same data can honorably reach different conclusions. Only those with access to the classified portions of the NIE and the classified internal review process can provide informed opinion on this point, and they are not legally permitted to do so.
The second inference could be disproved if yet newer information reveals that the "new" information underlying the change in judgment turns out to be wrong. Such information could appear at any time, and the public disclosure of the NIE summary could be the crucial act that makes it available to the IC. Indeed, Stratfor quotes Israeli defense minister Ehud Barak disagreeing with the substance of the NIE: "It seems Iran in 2003 halted for a certain period of time its military nuclear program but as far as we know it has probably since revived it" (emphasis added).
Virtually every decision must be made under uncertainty. It's inappropriate to wait until uncertainties are reduced to judge the merits of a decision that had to be made before the data were available. It's also inappropriate to criticize decisions that must be made under uncertainty because their outcomes cannot be predicted with certainty.
In the case of Iran's nuclear program, the key risk management issue is: How precautionary ought decisions be in the face of uncertainty? Reasonable people can disagree about this, and as Stratfor reports, it is entirely reasonable to expect Israeli risk managers to be more risk averse than those in the U.S. Unfortunately, little of today's reporting and commentary deals with the issues of whether risk management decisions with respect to Iran should be precautionary in the fact of uncertainty and how much risk is acceptable.
