Given its regulatory responsibilities over the American drug market, the Food and Drug Administration (FDA) performs a critical oversight function for US citizens and the wider world. Despite its importance, there are relatively few cost-benefit analyses of FDA’s regulatory decisions (Adler and Posner, 1999; Sunstein 2002, 2018). This has led to contradictory criticisms; while some argue the FDA unnecessarily delays the introduction of new drugs, others argue its regulatory decisions are expedited at the expense of consumer safety (Grabowski & Vernon, 1983; Olson, 2002, 2004, 2008; Dixon and Gagnon 2004).

Given such disagreements, Philipson et al. (2008) offer a much-needed assessment of a key program the FDA uses to assess new drugs, the Prescription Drug User Fee Acts (PDUFA). Designed to incentivize the timely review of new drug applications and supplement the FDA with funding Congress has been unwilling to provide, the PDUFA allows pharmaceutical companies to pay the FDA fees in exchange for guarantees the FDA will reach a decision on most new drug applications in six to ten months.Footnote 1

The welfare implications of the FDA’s decisions under PDUFA are potentially significant; the timely approval of new drugs influences pharmaceutical companies’ profitability by extending the length of time drugs are under patent, while expediting patients’ access to new treatments may extend their lives. However, PDUFA’s benefits must be assessed along with the costs of drugs the FDA approves but which subsequently have adverse effects and are removed from the market. Although some argue the FDA’s potential loss of PDUFA fees increases the odds that some harmful drugs will be approved (Carpenter et al., 2012), Philipson et al., (2008) find the quicker introduction of drugs under PDUFA I-II was beneficial, adding $7–11 billion to producer surplus and $7–20 billion in consumer surplus, a total welfare increase of $14–31 billion.

However, there are significant problems with the data Philipson et al. employ. To estimate the costs of drugs the FDA approved but subsequently had to withdraw for safety reasons, Philipson et al. use the FDA’s Adverse Events Reporting System (FAERS), which collects individual-level data from consumers, drug manufactures, and healthcare professionals who contacted the FDA because they believe a drug caused an adverse health event.Footnote 2 As FAERS offers individual-level data on drugs that may cause health problems, FAERS data have been used in other studies of PDUFA as well (e.g. Grabowski & Wang, 2008; Olson, 2008).

The principal problem with using FAERS data to estimate the costs of recalled drugs is that FAERS data do not measure the incidence of adverse health events in the US population, but only report information from people that contact the FDA because they believe a drug caused a health problem. Systematic reviews of spontaneous reporting systems have found 94% of adverse events for some categories of drugs go unreported, a finding that “raises concern about FAERS-based analyses” (Alatawi & Hansen, 2017, 4; Hazell & Shakir, 2006).

When appropriate data are used to measure the welfare costs of a single drug, Vioxx, which the FDA approved but was subsequently withdrawn after it was found to cause heart attacks, Philipson et al.’s findings are reversed. Instead of finding PDUFA I-II to be welfare enhancing, adding $14–31 billion in social surplus, there was a $1-201 billion reduction in social welfare during the period Philipson et al. examine. To illustrate the problems with using FAERS data in cost-benefit calculations, the rest of this paper is structured as follows: Section I summarizes Philipson et al.’s (2008) findings and discusses their use of FAERS data. Section II uses estimates from Graham et al. (2005) to recalculate the monetary costs of the deaths Vioxx caused. I conclude by discussing the problems statistically unlikely regulatory errors pose for the drug approval process.

1 Cost benefit analysis and the fda

In estimating the costs and benefits of the PDUFA, Philipson et al. (2008, 1306) develop a framework measuring the speed of drug approval and the costs and benefits of drugs the FDA approves.Footnote 3 In the seven-year period (1998–2005) Philipson et al. examine, the FDA withdrew 9 drugs from the market. I focus on one of these drugs, rofecoxib (Vioxx) that Merck introduced in 1999 to treat arthritis symptoms. After earning Merck $2.5 billion in profits, Vioxx was withdrawn in 2004 after it was found to increase the odds of heart attacks, a problem Merck understood prior to FDA approval (Thomas, 2006; Berman 2020).

To calculate Vioxx’s costs, Philipson et al. (2008, 1321) use the FDA’s FAERS database to estimate Vioxx caused 8,013 heart attacks requiring hospitalization and 1,349 deaths. However, given the self-reported nature of FAERS data the FDA warns against using FAERS for such purposes, stating: “FAERS data cannot be used to calculate the incidence of an adverse event or medication error in the U.S. population.”Footnote 4 To illustrate the problems with using FAERS data in cost estimates, the next section uses epidemiological data on the actual incidence of heart attacks and deaths Vioxx caused to recalculate the costs of PDUFA I-II.

2 The costs of vioxx

To calculate the costs of Vioxx, I use estimates from Graham et al. (2005) which studied the incidence of death among all Californian residents enrolled in Kaiser Permanente who were prescribed Vioxx between January 1, 1999 and December 31, 2001. The Kaiser data measures the incidence of heart attacks among patients prescribed Vioxx relative to patients that were prescribed other nonsteroidal anti-inflammatory drugs (NSAIDs). As the number of Vioxx prescriptions issued in the US is known, Graham et al. (2005, 480) estimate that Vioxx caused between 88,000 and 140,000 heart attacks requiring hospitalization, 44% of which were fatal. This indicates that Vioxx caused between 38,720 and 61,600 deaths, figures significantly larger than the 1,349 deaths Philipson et al. (2008, 1322) estimate from FAERS data.Footnote 5

Table 1 compares the number of heart attacks, deaths, and life years lost due to Vioxx depending upon whether one uses estimates from Philipson et al. or Graham et al. Using Philipson et al.’s (2008, 1322) estimate that fatal heart attacks caused by Vioxx reduced life expectancy by 16.2 years, and using Graham et al.’s lower-bound estimate that Vioxx caused 38,720 deaths, Vioxx caused a reduction of 627,264 life-years lost, a figure significantly larger than Philipson et al.’s (2008, 1321) estimate that all 9 drugs approved under PDUFA I-II that were subsequently withdrawn caused a maximum of 55,600 of life-years lost. However, if we use Graham et al.’s upper bound estimate that Vioxx caused 61,600 deaths, the total life-years lost as a result of fatal Vioxx-induced heart attacks rises to 997,920.

Table 1 Heart Attacks, Deaths, and Life Years Lost to Vioxx
Full size table

To calculate welfare losses requires estimating the number of deaths Vioxx caused, and then converting these life-years lost into monetary calculations. The reduction in life-years due to Vioxx is calculated using Philipson et al.’s (2008, 1322) estimate that the average number of life-years lost as a result of death caused by Vioxx is 16.2 years. Table 2 compares the minimum and maximum dollar values of the life years lost due to Vioxx depending upon whether one uses Philipson et al.’s or Graham et al.’s estimates, the 3% or 9% discount rates Philipson et al. employ, and Philipson et al.’s assumption that a year of life is worth either $100,000 or $300,000.

Assuming a life-year is worth $100,000 discounted at the maximum of 9% annually, and Graham et al.’s lower bound estimate that Vioxx caused 38,720 deaths and a reduction of 627,264 life years, the lower bound estimate of Vioxx’s monetary cost is $32,204,119,798. If we assume each life-year is worth $300,000 and use the lower discount rate of 3% annually, the monetary cost of the 38,720 deaths Vioxx caused is $146,050,319,078. However, if we use Graham et al.’s upper bound estimate that Vioxx caused 61,600 deaths, the total life-years lost rises to 997,920, and the cost of Vioxx ranges between $51,233,826,952 and $232,352,780,352 depending upon whether a life-year is valued at $100,000 or $300,000, and whether a 3% or 9% discount rate is used.

Table 2 Monetary cost of vioxx
Full size table

Table 3 presents total social welfare surplus calculations depending upon whether one uses Philipson et al.’s FAERS data or the Graham et al. fatality data. Philipson et al. (2008, 1318) estimate that the maximum social surplus that PDUFA generated, including both higher profits to pharmaceutical companies and higher welfare to consumers, was $31 billion. However, as Graham et al. estimate the welfare costs of the deaths Vioxx caused range between $32-$232 billion, even if we use the lower bound cost estimates from Graham et al. ($32 billion) and Philipson et al.’s upper bound estimate of the social welfare benefits PDUFA generated for both drug producers and consumers ($31 billion), there was a reduction in total social surplus of $1 billion. However, if we use Graham et al.’s upper bound estimates regarding the cost of the maximum number of deaths Vioxx caused ($232 billion), under PDUFA there was a reduction in total social surplus of $201 billion.

Table 3 Social welfare effects
Full size table

To summarize: using appropriate epidemiological data, Vioxx imposed welfare costs ranging from between $32 to $232 billion. Instead of supporting Philipson et al.’s conclusion that PDUFA generated a maximum welfare surplus of $14–31 billion, Vioxx is estimated to have caused a social welfare reduction of between $1 and $201 billion.Footnote 6

3 Conclusion

Although cost-benefit analysis aids the evaluation of public policies, such analyses must employ appropriate data when assessing agencies’ decisions. While Philipson et al. (2008) offer a sophisticated account of the tradeoff between the speed of drug reviews and the costs of drugs subsequently removed from the market, their use of FAERS data significantly underestimates the costs of the FDA’s decisions, as tens of thousands of heart attacks and deaths Vioxx caused do not appear in FAERS data.

Regardless of what caused the FDA to mistakenly approve Vioxx, this regulatory episode illustrates the challenges facing the FDA. Specifically, even if the FDA’s drug approvals are, on average, extremely accurate, statistically infrequent regulatory errors can have dramatic welfare effects if they involve widely prescribed drugs. Although regulatory failings such as Vioxx are infrequent, the billions of dollars in social surplus the FDA generates over years of accurate regulatory decisions can be wiped out by a single error.

In the case of Vioxx, the FDA’s error may have killed as many Americans as the Vietnam War, and the FDA will require years of accurate regulatory decisions to generate social welfare benefits to compensate for the costs of this single error. Regardless of the strategies used to assess the FDA’s review times, and comparisons of the number of recalled drugs before and after various PDUFA regimes, use of spontaneous reporting systems such as FAERS is inadvisable when estimating the welfare consequences of FDA regulatory decisions.