FormalPara Key Points for Decision Makers

Pharmaceutical companies had a tendency to over-estimate the likely expenditure of new medicines appraised by the All Wales Medicines Strategy Group (AWMSG).

Awareness of data relating to actual medicines expenditure after previous positive appraisals may help to better inform future estimates, and guide decision makers when considering their recommendations.

1 Introduction

The introduction of new, clinically and cost-effective medicines on the basis of pharmacoeconomic evaluation (or health technology assessment [HTA]) aims to improve health at a population level. However, the possible opportunity cost (i.e. in a health system with limited resources introduction of a new medicine may mean that other benefits will be forgone) must also be taken into account [1]. The National Institute of Health and Care Excellence (NICE) provides recommendations based on the clinical and cost effectiveness of new medicines to the National Health Service (NHS) in England and Wales (UK). Wales follows NICE single HTA guidance in relation to new medicines where available. However, in certain circumstances the All Wales Medicines Strategy Group (AWMSG) will appraise the clinical and cost effectiveness of new medicines for use in Wales [2]. AWMSG was established in 2002 as a statutory advisory, Welsh Assembly-sponsored public body under the 1977 NHS Act. Its remit is to provide advice on medicines introduction, optimisation and prescribing to the Welsh Government in an effective, efficient and transparent manner [3]. AWMSG will not normally appraise a product if NICE intends to publish guidance within 12 months of the date of market authorisation. However, when a medicine or indication is not on the future NICE work programme AWMSG may issue advice, although this is interim to NICE guidance should this be subsequently published [2]. The appraisal for a medicine is typically based upon the UK list price of the medicine excluding Value Added Tax (VAT). However, in line with the UK Pharmaceutical Price Regulation Scheme 2014 [4], companies are able to include a patient access scheme in their submission to NICE or a Wales Patient Access Scheme (WPAS) [5] as part of their submission to AWMSG. This allows a variation in the cost of the medicine from the list price, and is applicable to the specific medicine only in association with a positive recommendation.

As part of the appraisal process, pharmaceutical companies submit clinical and cost-effectiveness data to AWMSG. The company’s submission along with additional information (such as clinical expert opinion) is used by AWMSG to appraise the new medicine and to recommend to the Welsh Government whether or not it should be made available for healthcare professionals to prescribe for the indication requested. Part of the data requested from pharmaceutical companies is an estimate of the budget impact of the new medicine for the period following its introduction. These data form a key part of the evidence considered by AWMSG when making its recommendation and its accuracy may have significant implications for funding within NHS Wales. Primary (general practice) and secondary (hospital) care NHS services in Wales are delivered by seven health boards and one hospital trust. The health boards cover different geographical areas, and provide both primary and secondary care services. The hospital trust provides cancer services to the south east of the country, on behalf of the relevant health boards. Hospitals within the health boards and the hospital trust may purchase and supply medicines to patients directly. Prescriptions for medicines supplied in primary care are dispensed by independent community pharmacy contractors. Once an AWMSG recommendation is ratified by the Welsh Government there is a requirement for health boards to provide funding to enable access to the medicine within 2 months [6]. Increased demand for new medicines represents a significant challenge for health providers and payers. The cost of medicines appraised by NICE and AWMSG in Wales in the 3-year period from April 2013 to March 2016 accounted for £431 million [7]. Accurate estimations of the costs associated with new medicines may help commissioners to facilitate their introduction into care pathways and formularies.

The aim of this study was to compare the anticipated expenditure estimated by the pharmaceutical companies at the time of appraisal submission with the observed expenditure on medicines recommended by AWMSG for use in Wales.

2 Methods

Medicines appraised by AWMSG between May 2005 and December 2013 were identified for inclusion in this study. These dates represented the first availability of observed medicines expenditure data, and the availability of medicine expenditure data, allowing for 3 years’ data collection post-December 2013. Estimated and observed expenditure data were compared in each of the first 3 years post-AWMSG recommendation. Estimated expenditure was based on that submitted by the pharmaceutical company (Market Authorisation Holder) at the time of appraisal submission. After review and possible revision as part of the HTA process, this information is presented in the AWMSG Secretariat Appraisal Report (ASAR), available on the AWMSG website (http://www.awmsg.org). In some cases, aspects of this information are treated as ‘commercial in confidence’, and to preserve this commercial confidentiality the observed and estimated expenditure for all medicines are anonymised in this work.

Observed expenditure was for medicine cost only, based on the prices paid for the medicines by health boards in Wales (excluding VAT). Resource costs, such as those for administration, monitoring, diagnosis, infrastructure and staffing, were excluded. Observed expenditure was determined by combining primary and secondary care usage data, derived from dispensing databases, starting in the first financial quarter following ratification of a positive AWMSG recommendation. Some of the medicines were recommended on the basis that companies provided the medicine with a patient access scheme. These confidential discounts reduce the cost of the medicine below the list price. For medicines with associated patient access schemes, the observed expenditure cost was adjusted to remove the discount and convert the cost back to list price.

2.1 Medicines Included

Information available from dispensing databases did not include the indications for which medicines were prescribed. However, it was important to minimise the likelihood that observed expenditure was related to different indications other than the one being appraised by AWMSG. Therefore, medicines were only included in this analysis if they were ‘new’ (i.e. not being prescribed routinely prior to approval), or if they represented a new strength or new formulation of an existing medicine with exclusive use in the recommended indication. Conversely, any medicine for which there was likely to be routine use for other indications prior to the recommendation of interest was excluded from the study. The first medicine included in the study was ratified following appraisal in May 2005.

2.2 Data Sources

Estimated expenditure of medicines in each of the first 3 years post-ratification (i.e. three instances per medicine, one for each year) was based upon pharmaceutical company submissions to AWMSG. In some cases this information was ‘commercial in confidence’ so no medicine or company was identified by name in the analysis (although it was available to the researchers). Primary care medicines expenditure data were obtained from the Comparative Analysis System for Prescribing Audit (CASPA) (NHS Wales Shared Services Partnership; version 1.0.15.0). This database records all prescriptions issued for dispensing by community pharmacies and which are subsequently dispensed and submitted to NHS Wales Shared Services Partnership for pricing. This is the mechanism through which community (primary care) pharmacies are reimbursed for their dispensing costs. Secondary care (hospital) medicines expenditure data was obtained from the Medusa database in Wales. These data capture issues of medicines made via hospital pharmacy dispensing systems. All expenditure was measured in pounds sterling in the relevant year of data capture. List prices of medicines in the UK generally remain constant for the life of the product; therefore, no adjustment was made for inflation. In the UK, VAT is not charged on medicines dispensed in primary care, but is charged on hospital medicines purchases. Company estimates were exclusive of VAT, so VAT (at 20% of expenditure) was removed from the observed secondary care expenditure data obtained from Medusa. Primary and secondary care data were then combined to give an overall figure for observed expenditure in Wales. Observed expenditure was calculated annually for the first 3 years (on a rolling 12-month basis), starting in the first financial quarter following ministerial ratification of the AWMSG positive recommendation for the medicine.

2.3 Data Analysis

The median estimated and observed expenditure in each of the 3 years following AWMSG recommendation were compared. Because the data were not normally distributed, the Wilcoxon matched-pairs signed rank test was used. Spearman correlation analysis was used to determine whether there was a statistically significant association between the estimated expenditure (independent variable) and observed expenditure (dependent variable) in each of the 3 years following positive recommendation. To allow comparison with other published literature, the number of predictions falling within 40% and within 100% of the observed expenditure was also analysed. This was based on the percentage difference in estimated expenditure versus observed expenditure, calculated as estimated minus observed divided by observed expenditure. All data were analysed using GraphPad Prism version 5.04 for Windows (GraphPad Software, La Jolla, CA, USA).

3 Results

From May 2005 to December 2013, 176 medicines were appraised by AWMSG, of which 49 were eligible for inclusion in the analysis. Of the remaining 127 medicines, 24 were not recommended by AWMSG and 103 had evidence of usage for other indications prior to the relevant AWMSG appraisal. Table 1 lists abbreviated indications for the medicines. The sum of the total estimated expenditure for the 49 medicines was compared with the sum of the total observed expenditure; total expenditure was overestimated by 41% (year 1), 52% (year 2) and 62% (year 3).

Table 1 Abbreviated indications of the 49 included medicines
Full size table

Median estimated and observed expenditure data are illustrated in Fig. 1. There was a statistically significant difference between median estimated and observed expenditure in each of the 3 years post-medicine introduction (p = 0.03, p = 0.006 and p = 0.001, respectively; Wilcoxon matched-pairs signed rank test). Estimated expenditure exceeded the observed expenditure for 36 (73%) of the medicines in years 1 and 2 and 39 (80%) medicines in year 3. In year 1 the mean difference of estimated minus observed expenditure was £165,000 for the overestimated medicines and − £169,000 for the underestimated medicines. The corresponding mean overestimates and underestimates were £256,000 and − £172,000 in year 2 and £335,000 and − £317,000 in year 3. In total, the expenditure of 42 (of the total of 49) medicines was overestimated in at least one of the 3 years post-introduction, with 32 overestimated in all 3 years. The expenditure of 17 medicines was underestimated in a total of 37 instances (seven were underestimated in all 3 years, seven were underestimated in only 2 years and four were underestimated in only 1 year). When the total estimated expenditure was compared to the total observed expenditure (combining the 3 years of data for each medicine), 38 of the 49 medicines (78%) were overestimated and 11 were underestimated. Eleven medicines had zero observed expenditure in the first year, and this was also true for nine medicines in the second year and four in the third year. Eight of the 11 medicines which had zero expenditure in the first year had zero observed expenditure in the second year and four medicines had no observed expenditure in any of the 3 years.

Fig. 1
figure 1

Median (with interquartile range) of estimated and observed expenditure in each year post-recommendation (n = 49). *p < 0.05, **p < 0.01 (Wilcoxon matched-pairs signed rank test). Yr year

Full size image

The correlation between estimated and observed expenditure in each of the 3 years following AWMSG recommendation is shown in Fig. 2a–c. Spearman r values for estimated and observed expenditure were 0.61, 0.70 and 0.72 for the first, second and third years post-introduction, respectively (all p < 0.0001). The correlation between estimated and observed expenditure in each of the 3 years following AWMSG recommendation was also calculated for subgroups of the sample of 49 medicines. The medicines were grouped based upon both the cost per patient per year and the estimated number of patients treated in year 1 post-recommendation. Spearman r and associated p values for each subgroup and year post-recommendation are shown in Table 2.

Fig. 2
figure 2

Estimated versus observed expenditure for years 1 (a), 2 (b) and 3 (c) post-recommendation. The 11 medicines for which there was a zero observed cost in year 1, the nine medicines for which there was a zero observed cost in year 2, and the four medicines for which there was a zero observed cost in year 3 are not represented on the graphs (a–c, respectively), but were included in the statistical analysis

Full size image
Table 2 Correlation of estimated versus observed expenditure based on medicine unit cost or patient number
Full size table

In the first year post-ratification, ten of the 49 medicines were estimated to within 40% of the observed expenditure, 16 were estimated to within 40–100% of the observed expenditure and 23 were overestimated by greater than 100%. Figure 3 shows the accuracy of the estimates over time. Data were also analysed to determine whether there was a tendency for individual pharmaceutical companies to over- or underestimate predicted expenditure. Six companies submitted three or more medicines within our dataset of 49 medicines. For five companies the number of their medicines with overestimated expenditure exceeded the number of medicines with underestimated expenditure. However, one company underestimated the expenditure of three of their four medicines submitted for appraisal.

Fig. 3
figure 3

Accuracy of predictions for the 49 medicines included in the study

Full size image

4 Discussion

This study compared companies’ estimated expenditure with the subsequently observed expenditure of new medicines recommended by AWMSG and used in NHS Wales (UK). There were 49 medicines included in the analysis. Median estimated expenditure tended to be greater than observed expenditure, and there was a significant difference between estimated and observed expenditure in each of the 3 years after introduction of the medicines. Total expenditure was overestimated by 41% (year 1), 52% (year 2) and 62% (year 3). Expenditure estimated by pharmaceutical companies was higher than the observed expenditure in at least one of the 3 years, for 86% of the medicines included in the study. Whilst most companies tended to overestimate expected expenditure in their submissions, one company consistently underestimated, with three of its four medicines having higher observed than estimated usage.

A study of the accuracy of analysts’ forecasts for 260 medicines launched between 2002 and 2011 in the USA was conducted by Cha et al. [8]. Forecasts of peak costs were obtained prior to medicine launch, and updated over the lifetime of the medicine. The number of predictions falling within 40% of the observed expenditure was 42%. This compares with 18% seen in our analysis. However, the study of Cha et al. [8] was based on consensus peak cost forecasts, whereas our analysis utilised total annual expenditure and a single estimate from the applicant company. An analysis of estimated versus observed costs for 15 medicines receiving US Food and Drug Administration (FDA) approval in the USA was conducted by Broder et al. [9]. In their study pre-launch budget estimates were compared with actual budget impacts (25 estimates for 15 medicines). Using the approach of Cha et al. [8] to analyse the data from Broder et al. [9] showed that only 20% of estimates were within 40% of the observed usage, which was broadly comparable with our own findings. Estimates presented in all three studies showed a tendency to overestimate expenditure, with those reported by Cha et al. [8] being the most accurate. On average, Cha et al. [8] had approximately seven forecasts for each medicine (1700 estimates for 260 medicines), from which they took a consensus—this may explain why estimates were more accurate than those in our study [8]. An analysis in 2012 of NICE-appraised medicines used in NHS England gave results that were even more accurate than the Cha et al. [8] analysis; 70% of technologies considered had estimates within 40% of the observed costs [10]. One of the reasons for the high level of accuracy of this work may have been the exclusion of medicines with small eligible populations and also of those where dosing varied between patients. Broder and co-workers [9] found that the smaller the estimated population, the larger the overestimate of expenditure. Estimates for medicines with a small anticipated treated population might be more susceptible to small variations in patient number, which might easily be seen in clinical practice. If this were to occur for high-cost medicines, the effect on the difference between estimated and observed expenditure could be dramatic. Similarly, where dosing has a significant effect on cost, an additional variable must be considered, making accurate estimates more challenging than for medicines with a single-dose regimen. However, grouping of the medicines in our sample according to estimated patient number in year 1 post-recommendation resulted in variable correlation between estimated and observed expenditure. For medicines where the estimated year 1 patient number was between 1 and 20, Spearman r values for years 1 and 2 post-recommendation failed to reach statistical significance, whilst the association in year 3 had a Spearman r value of 0.62. The correlation between estimated and observed expenditure was stronger for medicines with an estimated treated population of between 20 and 100. However for medicines with a larger estimated treated population the Spearman r values were lower, again suggesting a weaker correlation. Taken together, this analysis suggested that medicines with an estimated year 1 treated population of between 20 and 100 showed the strongest correlation between estimated and observed expenditure. It must be noted that the grouping of medicines was somewhat arbitrary, having been designed to produce approximately equal numbers in each group. This, along with the relatively small size of each group (range n = 15–17) may limit any conclusions drawn.

Whilst our study included a relatively small number of medicines, the results suggested a general tendency by companies to overestimate expected expenditure, although one company underestimated the expenditure for three of its four included medicines. The tendency to overestimate may have been due to factors such as a more gradual uptake of the medicine than anticipated following recommendation, an intention by companies to present an optimistic prediction of usage in their submissions to AWMSG, the introduction of competitor medicines into the market, or price discounting (particularly in secondary care). Broder et al. [9] stated that pre-launch predictions of newly developed medicines tended to considerably overestimate observed sales (and companies may have a desire to maximise the media impact of pre-launch sales estimates). One of the implications of consistently overestimating expenditure could be that HTA bodies (if they take account of budget impact), and healthcare commissioners more widely, interpret the figures as a ‘worst-case scenario’ upon which to base implementation decisions. If the estimate was within an acceptable range, this could influence the HTA body to come to a positive recommendation, based on the assumption that expenditure was unlikely to exceed the proposed estimate. However, if the estimate were to be too high, this might affect the chance of positive recommendation even if a likely overestimate were to be taken into account. Conversely, proposing a smaller estimate of expenditure (which could ultimately be an underestimate) could make the medicine appear more affordable. This might be seen by companies as a way of increasing the chance of a positive recommendation.

Another explanation for the observed tendency to overestimate expenditure could be the data used in our analysis. Secondary care data were based on the price paid for the medicines by health boards, and any available discount would have been reflected in these data. It is also possible that observed expenditure may have been influenced by variation in the prices paid for the medicines between hospitals due to local pricing agreements. Cha et al. [8] suggest that there are differences by therapeutic area with central nervous system (CNS) and cardiovascular medicines being most commonly overestimated and cancer drugs being most commonly underestimated. In our study it was difficult to make any similar comparisons as the dataset was relatively small (49 medicines). However, analysis of the number of over- and underestimates by therapeutic area suggested that for most categories (cardiovascular, infections and diabetes mellitus) the expenditure was overestimated, although for CNS medicines (epilepsy, pain, schizophrenia, Parkinson’s disease and attention deficit hyperactivity disorder [ADHD]) four medicines were underestimated and four medicines were overestimated (data not shown). Due to inclusion of many cancer medicines in the NICE work programme, and consequent exclusion of the particular medicines/indications by AWMSG, the number of cancer medicines included in our analysis was too small to allow any generalisations to be made. Grouping the medicines by unit cost (measured as cost per patient per year) did, however, reveal an interesting trend. For medicines with a lower unit cost, the estimated and observed expenditure in each of the 3 years post-recommendation was better correlated than for the sample of 49 medicines taken together (having numerically higher Spearman r values). Furthermore, there appeared to be an inverse relationship between Spearman r value and unit cost, suggesting that as unit cost increased, estimates of expenditure were less well correlated with observed expenditure. This perhaps reflects the fact that if the unit cost is higher, any variation in the number of treated patients would have a more significant impact on expenditure.

One of the limitations of the medicine usage databases from which data were extracted was that the indication for which the medicine was prescribed could not be determined. Our study included only 49 medicines from a total of 152 that were recommended by AWMSG between 2005 and 2013. The decision to include a medicine was based on the pragmatic criterion of being able to establish likely usage for the indication recommended by AWMSG. Therefore, the analysis included a specific group of medicines being appraised for their first licensed indication. It is conceivable that estimated expenditure for the medicines not included in our analysis might have been more or less accurate than those included in our sample. Estimates for medicines being appraised for a further indication may have been more accurate due to the availability of historical expenditure data from existing indications on which to base the likely level of uptake. Such data would not be available for new medicines being appraised for their first indication. A further limitation was that we were unable, for confidentiality reasons, to disclose the identities of the medicines and pharmaceutical companies included in the sample. The provision of this information might have been of value to readers to identify medicines or companies where estimates tended to be consistently higher or lower than the observed expenditure, and therefore to inform future predictions. However, given the relatively small number of medicines included in this study, the generalisations that could be drawn from such an analysis are likely to be somewhat limited.

5 Conclusion

In this study, annual company-estimated expenditure was compared with observed expenditure from Welsh NHS prescription databases for 49 medicines, recommended over the period May 2005–December 2013. Whilst estimates appeared to correlate with observed expenditure, when combining the estimated and observed expenditure for the first 3 years following recommendation, 78% of submissions overestimated future expenditure. Appraising organisations such as AWMSG consider the estimated expenditure (as an indicator of affordability) of any new medicine when developing their recommendation. Thus, companies should try to ensure the estimate is as accurate as possible when submitting to HTA bodies. An awareness of the accuracy of previous predictions may help to better inform this component of the process of new medicines introduction.