Forecasting Economic Depression:
Illustrating the pitfalls of expert elicitation
30 Mar 2009 in Regulatory Science, Information Quality
In recent years there has been a notable increase in the use of expert elicitation in human health risk assessment. The method usually involves empaneling a group of experts and, through a carefully crafted and complex set of procedures, asking each panel member to provide a subjective probability that some phenomenon that cannot be directly observed is true or false. The Environmental Protection Agency has an informative external review draft white paper on the subject.
In environmental health, expert elicitation has been used to quantify the risk of cancer from drinking water disinfection byproducts, the likelihood that routine exposure to particulate matter in air causes premature mortality, and the magnitude of uncertainties related to climate change. Each is a tough scientific question. For example, the risk involved may be quantitatively small, and hence hard to discern, or the scientific uncertainties may be very large. Judgment is required, and the judgments of scientists inevitably reflect a mixture of scientific expertise and nonscientific opinion.
The need for discerning science from policy in expert judgment has been recognized for decades, at least since the 1983 National Research Council Red Book. No consensus yet exists concerning how to do this in practice. As a result, practitioners of expert elicitation typically acknowledge the problem but not much else. The EPA external review draft white paper mentioned above, for example, says that Agency technical support documents relying on expert elicitation should address "[p]ossible correlations with non-elicited components of the overall analysis or policy question" -- a phrase that, when translated into plain English, means the inflitration of experts' policy views into their characterization of science.
Today's Wall Street Journal has an example drawn from a very different arena -- macroeconomic forecasting -- that offers a wealth of insight about the problems with expert elicitation.
The article by Wall Street Journal reporter Justin Lahart (subscribers, nonsubscribers) probes the question, "How likely is it that the current economic recession will become an economic depression?"
To illustrate the variance in expert opinion, Lahart quotes estimates from (in increasing order of likelihood) Northern Trust's Paul Kasriel (1%), Harvard's Robert Barro (20%), Moody's Investors Service's John Lonski(30% in early March, 20% today). The average esttimate from a recent WSJ poll of economists was 15%.
Each of these economists has policy views; Barro, in particular, is not shy about expressing them. The important thing is not whether Barro's (or others') policy views are correct. Rather, what must be understood is that expert elicitation of economists forecasts about the likelihood of a depression contains both scientific and policy components. Where does their economics insight end and their policy views begin?
Lahart gives another reason for being careful in interpreting such forecasts: There is no fixed definition for an "economic depression." Barro's definition (which the WSJ used for its poll) is a decline in per-person economic output or consumption of more than 10%. But this (like any other) definition is arbitrary. Some economists asked to give their likelihood estimate may have different definitions in mind. Moreover, some economists polled by the Journal may well have based their estimate on a different definition than Barro's.
In environmental heath applications of expert elicitation, this phenomenon is called "scenario rejection." In its most obvious form, the participant simply refuses to give an answer. More likely, the respondent gives answers but bases them on assumptions and definitions different from those provided by the researchers. No one knows what to do about this, in large part because it is hard to detect -- and it is especially hard to detect when the expert participant does not want to be detected.
In macroeconomic forecasting, the act of making a prediction can influence the accuracy of the forecast. Markets incorporate forecasts of future conditions in valuation. Human health risk assessors do not have to deal with this problem; the act of estimating the likelihood that, say, a fixed dose of disinfection byproducts causes bladder cancer will not affect the true probability.
Expert elicitation of human health risks suffers from two particular problems that also arise among macroeconomists forecasting the likelihood of depression. First, elicitation schemes typically report the average value, just as the WSJ reported that the average in its poll of economists was 15%. Experts whose opinions are being elicited can influence the average estimate by exaggerating their own.
Second, experts often have a stake in the outcome of the elicitation exercise. In macroeconomic forecasting, every expert elicited has his own estimate. In human health risk estimate, the scientists who participate are themselves experts in the field and, unsurprisingly, they have strong intellectual commitments to what they have already published. It may be possible to predict the outcome of an expert elicitation simply by knowing the identities of the experts whose views are being elicited.
This is especially worrisome when experts (whether for human health risk assessment or macroeconomic forecasting) are selected to achieve a "balance" among political or policy "stakeholders." There is no role in science for political or policy stakeholders, so this design subverts the scientific integrity of the exercise at the outset.
