Objectivity in Risk Assessment:
The National Intelligence Estimate on Iran, Part 4

1 Feb 2008 in Regulatory Science, Regulatory Policy

The subject of the 2007 National Intelligence Estimate of Iran's nuclear weapons program is not the dominant subject of news reporting that it was when we first posted on it. However, a conventional narrative has developed to the effect that Iran's nuclear ambitions and developmental efforts are no longer a legitimate concern.

Today we hope to finish our series on this subject showing why this narrative is based on value-based preferences that various people and interest groups hold, and is not supported by the NIE itself.

As a refresher, here's what we've already covered:

An introduction to the document as a risk assessment, not a prescription for policy.
An explanation why, if it true, the document should be interpreted as good news regardless of how precautionary one believes U.S. policy ought to be with respect to managing this risk.
A categorization of recent news and commentary: (a) those who were skeptics of the 2005 NIE and are believers of the 2007 NIE because they oppose aggressive efforts to challenge Iran's nuclear ambitions; and (b) those who were believers of the 2005 NIE and are skeptics of the 2007 sequel because they support such efforts.

There are sound policy reasons why commentators would sort themselves this way based on the degree of precaution that they believe ought to guide policy with respect to the narrow issue of Iranian nuclear ambitions, as well as broader matters such as Middle East policy and the global challenge posed by imperial Islamism.

Having read much of the news and commentary (life is too short to have tried to read it all, much less succeed), and the text of the NIE summary again several times, a few new insights come to mind:

The 2005 and 2007 NIEs are not necessarily in conflict precisely because they are separated by at least two years' data.

According to the 2007 NIE, Iran continues without delay to enrich uranium that would be usable in nuclear weapons. The Intelligence Community now believes that Iran stopped work on the next steps -- certain aspects of weapons development -- in 2003. The significance of this "stop work order" depends on how quickly it can be rescinded and how much time was lost by delay. The consensus answer appears to be that these programs could be restarted quickly, and that the delay is not significant given the lead time necessary to build up an adequate inventory of highly enriched uranium.. Because the ultimate purpose of the NIE it to evaluate the prospects of Iran credibly fielding a nuclear weapon, these differences are small.

The two documents are similar in another respect. The 2007 NIE says Iran is susceptible to external pressures such as sanctions, and that its decisions follow a rough cost-benefit analysis -- meaning that its policies with respect to nuclear weapons are more utilitarian than ideological. The 2005 NIE said the Intelligence Community believed that Iran was "determined to develop nuclear weapons despite its international obligations and international pressure," but that it was not "immovable."

These statements might be inconsistent, but we do not read them to be so. All that is needed to reconcile the 2005 and 2007 positions is for either or both of the following statements to be true:

The costs to Iran to sustain its nuclear weapons program increased significantly since 2003.

This could be true if, for example, the financial cost of the project increased a lot, or previously was severely underestimated by Iran's military budgeteers. Cost overruns are commonplace all over the world, including the U.S. The pressure to understate cost could be much more severe in a theocratic state such as Iran.

Alternatively (but not mutually exclusively), the cost of externally imposed sanctions could have increased significantly. This need not be limited to just the effects of easily observable phenomena, such as U.N. resolutions. One important but routinely overlooked new cost is that since 2003 the U.S. has had in place an extensive, semi-permanent military presence next door in Iraq. It is common to think of that military presence in strictly "Iraq War" terms, but its indirect effects on Iran could be substantial.

Not all costs are quantifiable or readily reduced to monetary values. Not all costs are observable, and some are so observable that we don't see them.

The benefits to Iran from sustaining ts nuclear weapons program decreased significantly since 2003.

For a nation lacking nuclear weapons, the benefits of gaining them are hard to disentangle from the value of surprise. For many years, Iran falsely claimed that it had no nuclear weapons program. Since the existence of its program was revealed, it has made it a government policy to publicize certain aspects of the program but hide the rest. This practice is entirely consistent with rational, cost-benefit calculation. For the same reason that it is in our interest to reduce uncertainty about Iran's nuclear activities, it is in Iran's interest to increase it.

By 2003, however, the value of surprise was lost, and Iran's presumptive enemies and opponents were surely made better off. The loss of surprise leads adversaries to counter with their own strategies, both defensive and offensive. Knowledge that Iran was actively developing nuclear weapons undoubtedly led to behavioral change among its enemies and opponents (and probably even its friends). In one of our previous posts, we noted that information that reduces uncertainty almost always has value, even if it increases your estimate of risk. A higher risk estimate is not as bad as bearing a higher risk but not knowing about it.

We don't know if costs have risen or benefits have fallen, or both. This information might be contained in the classified portion of the NIE. Publicly disclosing it, however, would undermine the value of keeping the Iranians uncertain about what is known, and what information led the Intelligence Community revised its views, which is far more important than the mere fact that it did so. Still, the unclassified summary hints that costs have increased. It says that in the judgment of the Intelligence Community, "the [2003] halt was directed primarily in response to increasing international scrutiny and pressure resulting from exposure of Iran’s previously undeclared nuclear work." It is notably silent about which aspects of "international scrutiny and pressure" mattered most.

Uncertainty about the date by which Iran will succeed to develop a nuclear weapon is much more serious than uncertainty about whether it is (or isn't) now actively doing so now.

The NIE summary says the Intelligence Community has only "moderate" confidence that it can narrow down the date to a half decade (2010-2015). Our naive risk management view is that five years is a wide window of vulnerability to a potentially catastrophic event, especially when the experts' confidence is so limited. (The NIE's description of "moderate" confidence is the least informative of the three confidence descriptors.)

More information about the range of expert views within the Intelligence Community would have greater value than more precision about the central tendency estimate.

Central tendency estimates of risk are often very useful, and especially so for risks with both low probabilities and low potential aggregate consequences. But they are generally not what risk managers need when they face low probability risks with high potential aggregate consequences. In the case of Iranian nuclear weapons, the probabilities involved are relatively high ("moderate" confidence) and the potential aggregate consequences vast (Iran will develop a nuclear weapon within 3 to 7 years). Foreign and military policy decision makers are likely to be very highly risk averse with respect to these concerns. To make informed decisions, they will want to know, at a minimum, the range of expert judgments across the multiple agencies (and even sub-agencies and offices within sub-agencies) involved.

There are good technical reasons for being concerned that consensus central tendency estimates may not even reflect the true central tendency. A known flaw of expert judgment elicitation procedures (such as what appears to have been used to derive the NIE) is the propensity of players to engage in strategic behavior so as to influence the value of the "consensus" estimate. Disclosure to policy-makers (though perhaps not the public) of the distribution of views can be very helpful in discerning whether the reported estimate is a widely shared view or the consequence of averaging disparate views. Better expert elicitation tools are needed to discourage strategic behavior.

Differences between the two NIEs on the ultimate risk question of interest are minor.

The most important issue is not whether Iran is or isn't now developing nuclear weapons. It's the date by which Iran is expected to succeed. On this question, the two reports offer views that are only subtly different. The 2005 NIE said there was "moderate confidence" that Iran's development of nuclear weapons was "unlikely before early-to-mid next decade." In 2007, it said it had "moderate confidence that" success by 2009 was "very unlikely," but "sometime during the 2010-2015 time frame bit "Iran probably would be technically capable of producing enough [highly enriched uranium] for a weapon sometime during the 2010-2015 time frame." These predictions aren't materially different.

The 2007 NIE also makes the following statement, which we have not seen picked up in news reporting or commentary:

We continue to assess with moderate-to-high confidence that Iran does not currently have a nuclear weapon.

Given the ambiguous definition of "moderate confidence," for many decision-makers this may be most worrisome statement in the entire report.

[add a comment]