How Not to Estimate Benefits:
The case of Avastin
21 Feb 2008 in Regulatory Economics, Regulatory Policy
The Wall Street Journal editorializes today against a longstanding Food and Drug Administration policy that values the benefits of drugs for terminal cancer patients solely in terms of life extension. Some visual aids may help make the issues easier to understand.
The Journal editorial board writes (temporarily available to non-subscribers):
In clinical trials, Avastin demonstrated the longest reported "progression-free survival" for patients with advanced breast cancer. That means they live longer before their disease spreads or worsens. An initial study submitted to the FDA showed that Avastin in combination with Taxol (another cancer therapy) delayed the growth of tumors by about 11 months -- some five and half months longer than Taxol alone. Additionally, more than twice as many patients experienced significant tumor shrinkage.
In February, Genentech also released the preliminary findings of a more rigorous follow-up study, including the FDA's "gold standard" of randomized and placebo-controlled clinical trials. It again confirmed that Avastin improves progression-free survival, though the full results have not yet been made public.
In other words, dying patients live nearly twice as long on average before their disease gets worse, and maybe longer. It translates into an improvement in quality of life by delaying the onset of symptoms. But only in a few isolated contemporary cases has the FDA deemed progression-free survival as a relevant "end point" for approval. There's no reason besides the FDA's complacency and archaic procedures; a recent review by the agency's own Science Board concluded that "evaluation methods have remained largely unchanged over the last half-century."
In February, Genentech also released the preliminary findings of a more rigorous follow-up study, including the FDA's "gold standard" of randomized and placebo-controlled clinical trials. It again confirmed that Avastin improves progression-free survival, though the full results have not yet been made public.
In other words, dying patients live nearly twice as long on average before their disease gets worse, and maybe longer. It translates into an improvement in quality of life by delaying the onset of symptoms. But only in a few isolated contemporary cases has the FDA deemed progression-free survival as a relevant "end point" for approval. There's no reason besides the FDA's complacency and archaic procedures; a recent review by the agency's own Science Board concluded that "evaluation methods have remained largely unchanged over the last half-century."
To simplify, the FDA appears to consider life extension as the only benefit terminal cancer patients get from these drugs. The more a drug lengthens life, the greater its benefits are presumed to be. If a drug improves the quality of life, those benefits appear to be either ignored or given little weight.
To understand why such a policy is misguided, we start from the presumption that what the FDA is trying to do is mimic what terminal cancer patients would choose if they had the agency's scientific and medical expertise. (If FDA experts are not trying to mimic the preferences of cancer patients--for example, if they are substituting what they think terminal cancer patients ought to prefer--then conflict between the FDA and cancer patients is inevitable. The problem cannot be solved without taking away the FDA's authority to make these decisions.)
Figure 1 below A illustrates four alternative drugs for terminal cancer patients. The horizontal axis measures life extension. The vertical axis measures quality of life, as judged by patients themselves. All four drugs--A, B, C, and D--achieve identical amounts of life extension, so under the FDA's policy they have identical benefits.
Terminal cancer patients would disagree. First, note that for every additional month that they are alive, Drug B (blue) enables patients to have better quality of life than Drug A (magenta). Leaving aside the cost of the drug, they will always prefer Drug B to Drug A.
Patients might also prefer drug C (orange) to Drug A. For most of the additional months they have to live, Drug C gives them a better quality of life. And they'd prefer Drug D (green) to all of the others.
If the FDA faced a situation like this, and all it considered was life extension, then it would approve all four drugs. If it took account of quality of life, it might not approve Drug C even though patients likely would prefer it if Drugs B and D were not available. (Or alternatively, if Drugs B or D did not work for them, or for some other reason they could not take them. Not all patients will have the same "benefit curve," and biological responsiveness may be so variable that Figure 1 illustrates four different patients taking the same drug. For now, we set aside that aspect of the problem to make the rest of it easier to understand. We'll come back to it later.)
If conflict between the FDA's experts and terminal cancer patients is possible even under this very simple scenario, it gets severe when the choices are more complex.
Figure 2 below shows the case where a new Drug E must be evaluated, and Drug A is the only currently available choice. Drug E does not deliver as much life extension as the existing drug, and if life extension is the only benefit the FDA considers, then Drug E will not be approved. However, many patients would strongly prefer Drug E to Drug A. They will be very unhappy with the FDA's decision because it does not mimic what they would choose if they had the same scientific and medical information.
Figure 3 below intensifies the conflict by making the quality of life benefits of the new drug (now called Drug F) superlinear. In practical terms, Drug F enables patients to hold onto a better quality of life for a longer period before their conditions deteriorate. Like in the previous case, many patients would prefer Drug F to Drug A, and some will prefer Drug F to Drug E even if Drug F does not enable them to live as long. But if only life extension matters, the FDA will not approve Drug F.
Figure 4 below illustrates open warfare between the FDA experts and the patients whose interests they are supposed to represent. Drug G provides a better quality of life until the very end of life, so a lot of patients would strongly prefer it. But it extends life just a little bit less than Drug C, which patients already dislike because it imposes significant losses in quality of life as the price for life extension. If the FDA considers only life extension as a benefit, it will not approve Drug G and it will be hated by terminal cancer patients.
A useful reform at the margin would be for the FDA to redouble its efforts to mimic the preferences of terminal cancer patients, who care a lot about how well they live as well as how long.
Ultimately, it isn't at all clear why the FDA should be in the business of making risk-benefit judgments on behalf of terminal cancer patients. No matter how much effort it devotes to estimating the array of things terminal cancer patients really care about, it will fail to fully capture everything that matters. If ever there was a subpopulation for which the FDA could leave decision-making entirely to patients themselves, surely this is it.
Return finally to the issue of patient variability. All of these curves could accurately and precisely reflect the average response across patients, and at the same time mask huge disparities in individual response. Even if one drug is clearly inferior on all margins for the average patient, it may be a superior choice for some patients.
Averages may be appropriate for government decision-making where individual risks are small. Averages become increasingly suspect as individual risks rise, and especially in cases like terminal cancer where there is neither an efficiency nor an equity argument for denying the actual people involved the ability to make their own informed choices.
