Are Businesses More Likely to Produce Biased Research than Government Agencies or Nongovernmental Organizations?
Virginia offers an example
23 Aug 2006 in Information Quality, Peer Review
We have posted on the matter of whether scientists affiliated with industry should be automatically excluded from federal peer review panels irrespective of the scope, scale or significance of their affiliation. The Center for Science in the Public Interest says any financial tie (a term it does not clearly define) with any industry constitutes a "potential" conflict of interest, and any financial tie of any magnitude with any industry or group that might be directly affected constitutes a "direct" conflict of interest.
We expressed concern about the breath of these proposed automatic exclusions. CSPI's stated justification is that industry funding may result in bias, and one commenter on our post asserted that "the possibility of bias in industry funded work is greater than in other work since I believe that financial incentives are greater than non-financial ones." But neither CSPI (nor the anonymous commenter) appears to be concerned about financial ties with regulatory agencies, activist groups and their supporting foundations irrespective of scope, scale or significance. If CSPI's conflict of interest rule were to apply, only for-profit financial interests would be grounds for exclusion. Thus, CSPI would succeed in automatically excluding its opponents and thereby silencing views contrary to its own.
The focus on financial interest alone is misguided. Serious conflicts can occur because of political or policy interests without the exchange of money. An example from the Commonwealth of Virginia illustrates our point. In this case, an ostensibly neutral traffic study was managed by senior political aides to the governor. They apparently hid their involvement until exposed by an activist with an opposing view.
Washington Post reporter Amy Gardner says senior staff working for Virginia Governor Timothy Kaine "personally managed a traffic study" performed by Virginia Department of Transportation (VDOT) engineers. A previous story by Gardner treated the study as a neutral analysis predicting "gridlock," an additional "300,000 car trips per day by 2025," and stop-and go conditions throughout the area that "could span six hours each day."
The study was inserted at the last minute into a local zoning controversy, but the involvement of senior political officials in the Kaine administration was not mentioned. Gardner noted, apparently with innocent irony, that the study was "well-timed" for Kaine given his declared plans for overhauling transportation policy in the Commonwealth.
Apparently, the study's timing was no accident and the objectivity of its content is in dispute.
Governor Kaine generally opposes additional population growth in Loudoun County, an outer suburb of Washington DC that is noted for its rapid population growth over the past decade and consequently increasing traffic congestion. Kaine's February 2006 transportation plan promotes "uniform traffic impact statements" as a provision to "improve coordination of land use and transportation." Presumably, these impact statements would be objective analyses of the likely effects of new housing developments on regional traffic.
Gardner reports that e-mails show Kaine's office was in fact deeply involved in the study though its role was hidden from view. According to Gardner, "Pierce R. Homer, the state transportation secretary, personally edited the study in part to make the traffic impact more meaningful to the public." Kaine's chief of staff William H. Leighty also was involved in some fashion with the report. Brian Roherty of the Right Growth Policy Institute obtained the e-mails under Virginia's Freedom of Information Act.
Despite previous denials by Kaine spokesman Kevin Hall, the personal involvement of senior political officials in the study is no longer in dispute, now that e-mails proving it have been disclosed:
Homer and Dennis C. Morrison, VDOT's Northern Virginia district chief, confirmed the general accuracy of the e-mails and acknowledged that the study was directed by senior Kaine administration officials and was not simply a VDOT initiative, as characterized by Hall last month.
In federal practice, analyses prepared or closely supervised by political officials are generally regarded as presumptively not objective. At a minimum, a rigorous independent peer review is required to ensure that political concerns did not bias the work product. Analyses performed by career agency staff also are viewed this way if the agency the analysts work for has a clear opinion or policy view on the subject. In this case, the only evidence of oversight was that which political officials in the Kaine administration performed.
The idea for this particular traffic study — the first one ever performed by VDOT — came from an environmental group that opposes the Loudoun County development and a former board member of the group who now serves as Homer's deputy:
Homer said he came up with the idea after a request from Christopher G. Miller, president of the Piedmont Environmental Council, to study Dulles South. The PEC is an active advocate in Loudoun land-use politics and stands firmly against development near the airport. A former member of its board, Scott Kasprowicz, is Kaine's deputy secretary of transportation. Kasprowicz participated closely in the oversight of the study, the e-mails show.
According to Gardner, Virginia Transportation Secretary Homer said "the study was a good idea no matter where it came from." In principle this is true, for public decisions based on more and better data and analysis could be better informed decisions. And the study could provide an objective characterization of the regional transportation impacts of the development in question.
However, the procedures followed in this case demonstrate a clear conflict of interest n the part of senior government officials and an allied stakeholder. Both the party making the request (Miller) and the overseeing political official (Kasprowicz) are affiliated with an advocacy group that has a significant stake in the results of the report. Given their implicit and explicit direction, VDOT engineers would have powerful reasons to understate the benefits and overstate the costs of the development. In her initial story, Gardner quoted a supporter of the development describing the study as "a partial assessment" because it failed to account for $200 million of road improvements offered by one of the developers. That is precisely the kind of error that a conflict of interest would be expected to produce.
Kaine's spokesman Hall says studies analyzing the regional impact of local development decisions are appropriate governmental functions that are now required by law:
"There's an overwhelming legislative mandate for this kind of information being made to local officials," Hall said. "The fact that the chief of staff convenes a meeting to discuss what's next is not surprising and doesn't show any kind of undue influence."
We agree on both counts, but note that Hall's characterization omits the pertinent facts. First, the legislative mandate is surely for objective traffic analyses — not work products honed to buttress the governor's political viewpoints. Second, governors' chiefs of staff convene meetings for other purposes than merely collecting information about numbingly technical studies are scheduled to be released — such as, for example, advancing the programmatic objectives of the governor.
There are useful analogies between this case and the conflict of interest claims recently made by the Center for Science in the Public Interest, about which Neutral Source previously commented. CSPI proposed to exclude from federal peer review panels scientists who have been funded by industry in even the most trivial manner. Combined with its database of presumably tainted scientists, we were troubled by two implications of their argument:
- Industry-funded research conducted by the named scientist is biased or otherwise substandard, without regard for its substantive quality.
- All other research conducted by the named scientist is biased or otherwise
substandard solely because the scientist received industry funding on other
projects, also without regard for its substantive quality.
Because CSPI's proposal is so Draconian, it would systematically exclude scientists affiliated with industry for reasons unrelated to the quality of their work. The operating principle would be an irrebuttable presumption that any industry affiliation whatsoever ensured scientific corruption. CSPI displayed no apparent concern about conflicts of interest among scientists affiliated with regulatory agencies, nongovernmental organizations (such as CSPI), or the network of foundations that liberally support them.
The Kaine administration's traffic analysis for the proposed Loudoun County development illustrates how regulatory agencies and nongovernmental organizations can be fatally infected by conflicts of interest, even when no financial ties exist. A financial relationship is simply unnecessary for a genuine conflict if political and policy incentives are strong enough. They appear to have been more than sufficient in this case.
From Neutral? on 26 August 2006, 11:00
An interesting anecdote, certainly supporting your arguments that bias does exist on both sides of these debates and that policies with absolute restrictions are misguided. I agree on both of these points.
Of course the anecdotes of industry funded studies being biased are plentiful and the weight of the anecdotes points to greater bias in this direction (I also agree that a good study on bias would be interesting). For good anecdotes in the industry funded direction, see any of numerous works by Wendy Wagner or David Michaels for examples.
From Editor on 22 September 2006, 10:30
The fact that anecdotes are plentiful, but empirical evidence is lacking, reinforces the need for ways to test content for bias. A non-rebuttable presumption of bias based on the identity of a source is a practical dead end. All sources are potentially biased in the eyes of someone.
OMB has established peer review as a favored tool for discerning the absence of material bias. According to OMB, influential information that has been peer reviewed has "adequate" objectivity, an inference Neutral Source would diplomatically characterize as a triumph of faith over practice.
We are more fond of OMB's transparency and reproducibility tests. When transparency and reproducibility are lacking, it is a signal that bias may be lurking underneath. In general, sources unwilling to show their work ought to be regarded as materially biased.
Transparency and reproducibility foster competitive review by competent third parties. In our view, it is more efficient and more effective to rely on opposing stakeholders to uncover bias than to rely on distinterested and anonymous peer reviewers.
Does Neutral? agree?