Journal Peer Review and Objectivity:
The case of Avandia (rosiglitazone) and the New England Journal of Medicine
29 May 2007 in Regulatory Science, Information Quality
We've been a consistent supporter of high standards for information quality, especially for federal agencies that disseminate influential information in the support of regulatory objectives. We've also been consistently concerned about one particular aspect of the information quality and peer review guidelines issued by the Office of Management and Budget: the rebuttable presumption that scientific, technical, economic or statistical information published in peer reviewed journals satisfies the guidelines' standard for objectivity.
In the past week, the New England Journal of Medicine published online an article by Steven Nissen and Kathy Wolski claiming that Avandia, GlaxoSmithKilne's blockbuster drug for type 2 diabetes mellitus, "is associated with a significant increase in the risk of myocardial infarction [heart attack] and with an increase in the risk of death from cardiovascular causes that had borderline significance."
The Nissen & Wolski study has received enormous press attention. In today's Wall Street Journal former Food and Drug deputy commissioner Scott Gottlieb asserts that the NEJM has political motives for subjecting the study to inadequate peer review and publishing the study with excessive fanfare. Though not directed at federal information quality law and policy, Gottlieb's commentary suggests circumstances when OMB's rebuttable presumption of objectivity ought to be replaced with a rebuttable presumption of bias.
In Gottlieb's commentary (available for a limited time to nonsubscribers), he makes the following allegations about NEJM:
- NEJM "rushed" the Nissen & Wolski study onto its website "to get ahead of the Food and Drug Administration's more careful evaluation of the same issues." [The Nissen & Wolski study is here.]
- NEJM's purpose was to "[p]aint[] the FDA as impotent, in order to argue for legislation winding through Congress that would increase regulatory hurdles for drug approvals," with these motives "made bare by its own editorial on the matter." [The editorial by Bruce Psaty and Curt Furberg is here. A contrary unsigned editorial published in The Lancet, a competitor of NEJM, is here.]
- NEJM (and apparently Nissen) intentionally blindsided the FDA; it "wasn't even given a heads up about the study's publication or its findings."
- Nissen (and possibly NEJM) did alert Rep. Henry Waxman, chairman of the House Committee on Oversight and Investigations. Waxman "seems to have known in advance that it was coming." On May 21, the same date the Nissen & Wolski study was published online, Waxman scheduled an oversight hearing "aimed at probing the FDA's 'handling' of the drug safety issues." [A link to the oversight hearing is here.]
- As further evidence that NEJM and/or Nissen were deeply engaged in the politics of drug regulation, "[t]hree days after the study was submitted to NEJM, and before it was published, the FDA commissioner received a letter about Avandia from members of the House Energy and Commerce Committee that seemed to reference the NEJM study." [We have not been able to locate a copy of this letter.]
So when editors pursue a political agenda, it's public health that pays a price. Degrading an institution that doctors depend on for balanced analysis and fair-minded editorial judgments isn't good for anyone.
Gottlieb identifies several technical flaws and limitations in the Nissen & Wolski study, but says NEJM minimized them when drawing inferences consistent with its political agenda:These shortcomings didn't stop NEJM from spinning the preliminary data into the conclusion that Avandia was a troubled drug mishandled by the FDA and comparing it to Vioxx, the pain drug withdrawn by Merck two years ago because it increased risk for heart attacks. Vioxx "represented a similar regulatory failure to insist on large trials of public health importance in a timely fashion," NEJM's editorial states.
Absent was any discussion by NEJM of the drug's benefits, or advice from diabetes experts on how doctors should counsel their patients based on the information.
Absent was any discussion by NEJM of the drug's benefits, or advice from diabetes experts on how doctors should counsel their patients based on the information.
Gottlieb is not alone. Physician bloggers have picked up on the extent to which this incident exposes bias in some medical journals; see, e.g., Sue Hughes at Medscape, Musings of a Distractible Mind, Notes from Dr. R.W., Kevin M.D. Musings points out that heart attacks are a major complication from diabetes, so teasing out any increase from the drug would be very challenging:
When you are dealing with a disease where one of the major complications is heart attack, the possible causes for increased heart attack are numerous. Plus the fact that good diabetic control is universally accepted as the standard of care makes using a true placebo in any trial with a diabetic drug unethical. While the overall increase in heart problems appears great, the absolute numbers are not so impressive.
Musings appears to share Gottlieb's concern that public health is the loser when medical journals get into politics and the net benefit of pharmaceuticals to patients is ignored:
If this scare causes people to back off of their diabetes drugs and hence worsen their control, it will have done harm. Poorly-controlled diabetics are more likely to develop complications (many of which are life-threatening) than those who are well-controlled. The question in this case is whether the harm done to these patients is greater or less than the potential harm of taking the drug in the first place.
Federal information quality law and policy have significant ramifications for what FDA can and cannot do in response to Nissen & Wolski. The coincidence of Nissen's ongoing congressional advocacy and NEJM's editorial (especially in contrast with The Lancet editorial) provide persuasive evidence that Nissen & Wolski is not entitled to the presumption of objectivity provided in OMB's information quality guidelines and peer review guidance. As a matter of law, therefore, FDA cannot presume that Nissen & Wolski is an objective characterization of cardiac risks. They must undertake their own rigorous analysis, which is made much more difficult because of how Nissen and NEJM have politicized it.
Gottlieb's commentary does not concern information quality, but it identifies obvious places to look for evidence rebutting OMB's presumption of objectivity. For example, the presumption should be rebutted automatically if, like in this case, the journal publishes a companion, message-reinforcing editorial. That's evidence of actual bias, and bias is incompatible with objectivity.
The presumption of objectivity also should be rebutted if the journal has a discernible political agenda consistent with an article's explicit or implicit policy recommendations. That's evidence of likely bias.
In these instances, the rebuttable presumption of objectivity should be replaced with a rebuttable presumption of bias. An agency that wants to disseminate the information would then have the duty of showing that, despite the actual or likely biases afflicting the journal in which an article was published, the scholarship in the article satisfies the objectivity standard.
PRIMARY DOCUMENTS
- Nissen SE and Wolski K, "Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes," New England Journal of Medicine, (Published at www.nejm.org May 21, 2007 (10.1056/NEJMoa072761) (PDF).
- Psaty BM and Furberg CD (editorial), "Rosiglitazone and Cardiovascular Risk," New England Journal of Medicine, (Published at www.nejm.org May 21, 2007 (10.1056/NEJMe078099) (PDF).
- Unsigned Editorial, "Rosiglitazone: seeking a balanced perspective," Lancet (Published online May 23, 2007 DOI:10.1016/S0140-6736(07)60787-9).
- Waxman H. "Chairman Waxman Announces Hearing on FDA’s Role in Evaluating Safety of Diabetes Drug," May 21, 2007.
