1 Introduction

Financing of novel medical technologies has increasingly become a fundamental policy issue for budget-constrained health care system. Accordingly, several countries have introduced mechanisms to control spending and align the acquisition of technologies to national health care priorities, such as equity of access and system sustainability [1].

In such regulatory and policy environments, newly developed medical technologies must be able to demonstrate not only their potential for improving health outcomes, but also their value for money, or cost-effectiveness. For technologies that are both more expensive and more effective compared to their relevant alternatives, cost-effectiveness is often measured as the additional cost that is required to achieve one incremental unit of effect. This ratio is then compared to a cost-effectiveness threshold that represents payers’ maximum willingness to pay (wtp) for that additional improvement in health. Therefore, a technology is considered cost-effective if it shows to generate more health in the target population at a lower cost than the maximum wtp of payers. In some countries, the cost-effectiveness threshold is explicitly stated, such as for example the £20,000–30,000 threshold used by the NICE in England for technologies undergoing a single or multiple technology appraisal. In other cases it remains implicit in the negotiations between technology providers and payers [2].

For manufacturers, the early consideration of cost-effectiveness into decision processes may provide insights on the potential impact of new business propositions and inform subsequent strategic choices alongside the life-cycle of the technology [3]. For example, cost-effectiveness may be used to define the maximum achievable price of the technology, conditional on its expected impact on health outcomes, or in other words, the price at which the incremental cost-effectiveness ratio is lower than payers’ wtp threshold [4].

However, anticipating what will be the likely cost-effectiveness of a device early on in its product development is not trivial. Many parameters on the mechanism of action of the technology and how these will affect patient-relevant outcomes may be estimated with uncertainty or even be completely unknown. Therefore, decision analytical models are often used as they provide an explicit framework to collate all the available evidence from different sources, and make transparent and explicit assumptions on unknown parameters. In addition, models can be used to extrapolate long-term effects on patient-relevant outcomes, and to consistently translate the uncertainty in the model parameters into a measure of the overall uncertainty over the decision of adopting the technology.

Particularly, for chronic diseases, Markov models are commonly used. In these models, patient’s conditions are modelled as a series of transitions between defined clinically relevant health states. Each state is associated with measures of outcomes (e.g. quality adjusted life years, QALYs) and costs, and the overall cost and consequences are calculated by considering the overall time spent in each state over a pre-defined time horizon [5, 6].

As part of the European Innovation Programme on Active and Healthy Ageing (EIP on AHA), the Monitoring and Assessment Framework for the European Innovation Partnership (MAFEIP) is a tool designed to support evidence-based decision-making processes for all institutions and users in the health and care sector [7]. In short, the MAFEIP is a flexible, web-based tool that, through a guided procedure, allows users to build their own economic evaluations based on Markov models. The MAFEIP tool is now fully functional and the Commission has required its use to evaluate the impact of the initiatives and projects submitted within four European Horizon 2020 calls [8,9,10,11]. Further details on the MAFEIP tool and its uses are available elsewhere [12].

Despite the relevance of the tool, there is a dearth of published studies showing empirical applications of MAFEIP and/or providing a critical assessment of its usability. The aim of this paper is twofold. First, by using a case-study on a fall-prediction device for elderly patients with orthostatic hypotension we aim to demonstrate how MAFEIP can be used to inform manufacturers strategic choices on product development based on a cost-effectiveness criterion. Secondly, we critically appraise the tool and suggest further improvements that may be needed for a larger-scale adoption of MAFEIP within and beside the EIP on AHA initiative.

2 Methods

In this study, an early economic evaluation using the MAFEIP tool is conducted to estimate the expected effectiveness and cost-effectiveness of the device, and thus to inform strategic choices that are required to bring the device from its early stages, to more mature stages until market access. Particularly, the results of the model are used to estimate the maximum achievable price of the device considering different payers’ willingness to pay thresholds. In the discussion, a critical appraisal of the characteristics of the MAFEIP tool is provided with recommendations of key aspects that should be integrated to make the tool more suitable for evaluating technologies in their early phases of development.

3 Case study on fall prediction devices

Orthostatic hypotension (OH), also called standing or postural hypotension, is a common and disabling condition in which significant drops in blood pressure occur when standing up from sitting or lying down. OH is associated with a series of symptoms including dizziness, light headed, blurred visions and may even cause fainting. Studies also suggest that OH may be positively associated with an increased risk of falling in older subjects, although evidence is still scarce and often reports contradictory results [13,14,15,16,17,18]. Falls in elderly subjects have serious consequences in terms of increased mortality and reduced quality of life [19], and over one-third of older adults are expected to experience at least one fall or more each year [20, 21]. In addition, management of fall-related injuries also put financial pressure onto budget constrained healthcare systems [20].

In recent years, there has been a surge in the development of wearable devices that enable monitoring of physical activities and behaviours, as well as physiological and biochemical parameters of the patients during their daily activities [21]. Therefore, wearable sensors are potentially appropriate to be used for chronic conditions and preventive purposes, as in the case of falls prevention monitoring in elderly patients.

The present case study is about an innovative wearable device under development that uses electrocardiogram (ECG) and short-term heart rate variability (HRV) to predict sudden drops in blood pressures (BP) due to OH. Devices using ECG/HRV to predict drops in BP present advantages over devices directly measuring BP. First, continuous measuring of ECG/HRV is less intrusive for patients than measuring BP. Second, BP measures, are not as stable and reliable as ECG/HRV. Third, ECG data also provide valuable information for the monitoring of other conditions that are prevalent in the elders such as cardiovascular diseases. In its preliminary design the device is intended to warn individuals and caregivers about an imminent risk of falling, thus prompting immediate preventive actions. The proof of concept for the device, i.e. the predictive ability of HRV signals to detect blood pressure drops due to OH has been shown in a previous study by Sannino et al. [22, 23]. The authors developed a mathematical model that correctly predicted drops in blood pressures with an error below the measurement error of sphygmomanometer digital devices (± 4.5 mmHg), a false negative rate of 7.5% and a false positive rate of 10%. Technical specifications of the model can be found in the patent application no. PCT/GB2015/052581.

4 Model structure of the MAFEIP and parameters input values

The MAFEIP tool allows to build Markov models up to five disease states. In this case study, a 3-state Markov model was used including a baseline state, which represents the baseline health condition of the target population, a disease/impairment state, which reflects the health state of subjects who experience the condition of interest (e.g. an acute trauma due to a fall), and a dead state (Fig. 1). In the present model, subjects start in the baseline healthy state and can either die from all-cause mortality or experience a fall. If a fall occurs, patients may require clinical management for moderate or severe injuries and experience a higher risk of death. After the fall, subjects that have not died return to the healthy state. Each state in the model is associated with a cost, and a measure of the health-related quality of life expressed in QALYs.

Fig. 1
figure 1

Schematic representation of the Markov model in the MAFEIP

Full size image

Model parameters, including costs and QALYs for each states and transition probabilities across states were derived from the published literature and are reported in Table 1. Nonetheless, at the time of the analysis, no clear evidence was found on the rate of falls that may be due to sudden drops in blood pressure after standing. Therefore a formal expert elicitation was conducted to obtain an estimate for this parameter together with its uncertainty. Costs were converted to UK £ at 2016 price levels using the UK government GDP deflator indices [24].

Table. 1 Parameters used in the model
Full size table

Since, no standard healthcare technology has been identified for imminent falls prediction and prevention, the device is compared to the standard of care, i.e. to the acute management of fall-related injuries whenever they occur. In the baseline analysis, it was assumed a cost for the device of £230 and an average lifetime of 10 years, after which replacement would be needed. It must be noted that the MAFEIP tool does not allow to model recurrent costs over specific time intervals (while it allows to model per-cycle recurrent costs). Therefore, the overall discounted acquisition cost for the technology has been considered as a one-off expenditure of £600 at the beginning of the model. This necessary arrangement ultimately overestimates the overall cost of the technology, as it implies that the whole cohort will re-buy the device over time regardless of whether patients are alive or dead. Nonetheless, due to the relatively low cost of the technology, and the effect of discounting, changes in the cost-effectiveness results are considered modest.

The relative risk of falling with the device was calculated using the baseline probability of falling (regardless of whether subjects had OH or not); the elicited percentage of falls that are due to OH; and the sensitivity of the device as reported in the study from Sannino et al. [22]. The effects on cost-effectiveness of variations in either the price of the device or its relative effectiveness in avoiding falls were explored by conducting deterministic sensitivity analysis using the MAFEIP. The perspective adopted for the analysis is the one of the health care system, and therefore only direct health related costs and consequences are considered. The model estimates the potential effects of using the intervention over a time horizon of 30 years, using discrete time cycles of 1 year. Lastly, a discount rate of 3.5% was applied to both future costs and consequences.

5 Results

In the baseline case, adoption of the device produced an estimated gain of 0.035 QALYs per patient and incremental costs of £ 518. The estimated incremental cost-effectiveness ratio is £ 14,719, which represents the additional cost that is required to generate a health gain of 1 QALY when using the device in the target population. Additional graphic outputs of the MAFEIP for the baseline case, including the cost-effectiveness plane and patients transitions across states are also reported in Appendix A.

Model results provide insights on which is the maximum price that could be charged at specific willingness to pay thresholds. For example, in the baseline case, the price for the device of £230 every 10 years, would be acceptable to payers, only if their willingness to pay for any additional QALY gained would be no less than the incremental cost-effectiveness ratio. Simple sensitivity analysis in the MAFEIP can help to identify which prices would be considered acceptable at specific willingness to pay thresholds. For example, Fig. 2A shows the variations in cost-effectiveness that result from a change in the price of the device to £130 every 15 years, and £300 every 6 years. Similarly to the baseline case, acquisition costs of the device in the sensitivity analysis have been modelled as a one-off costs at the beginning of the model, and considered equal to £200 and £1,000 respectively, that is approximately the whole discounted cost of the technology over a time horizon of 30 years. At a WTP threshold of £20,000, charging the highest price would likely lead to a refusal by the payer, since the resulting incremental cost-effectiveness ratio would be higher (£26,000). Similarly, Fig. 2B shows that if the true incidence of falls in the intervention group was slightly higher than the expected value used in the baseline case, payers would be unlikely to adopt the device. Specifically, even at a £20,000 threshold, an increase in the incidence from 0.26 to 0.28 would make the device not cost-effective. However, keeping fixed the baseline probability of falling and the percentage of falls due to OH, to get this increase in incidence, the sensitivity of the device should be lower than 0.45, which is quite unlikely.

Fig. 2
figure 2

Deterministic sensitivity analysis using MAFEIP on costs (Panel A) and expected incidence of falls with the intervention (Panel B). Results compared to a willingness to pay threshold of £20,000 per Qaly. The orange dot in the picture marks the intersection of the sensitivity line with the dashed line representing the willingness to pay threshold (here £20,000 per QALY). This point splits the sensitivity line in two segments, a right segment where the technology is cost-effective and a left segment where the technology is not cost-effective

Full size image

6 Discussion

This study reports a case study of an economic evaluation of a fall-prevention device in its early phases of development. The MAFEIP tool was used to build a Markov model and to calculate the expected costs, effects, and cost-effectiveness of the device. Further sensitivity analyses were also performed to estimate the probability of being cost-effective at different prices of the device and payers wtp thresholds.

Early economic modelling can provide useful insights to manufacturers, informing the economic viability of their device early in the product development phases, and informing strategic considerations such as pricing, further product developments. However, one of the key challenges of conducting early-stage economic evaluations of medical technologies is that, by the time of the assessment, there may not be solid evidence on some of the parameters required in the model. For example, in this case study, the proportion of falls due to OH was mainly unknown, and had to be estimated through expert elicitation. However, rather than providing a reliable estimate of cost-effectiveness for the technology under assessment, the ultimate aim of conducting early economic models is mainly to explicitly characterize the existing uncertainty given the available evidence, to assess its relevance for decision making and ultimately to support decisions on what could be done to reduce it. In fact, even simple univariate sensitivity analysis through the MAFEIP tool can inform which parameters in the model has the highest impact on the expected cost-effectiveness of the device, providing initial insights on where further clinical development and evidence generation should focus.

Overall, the MAFEIP tool allows to quickly create early economic models, and to explore model uncertainty by performing deterministic sensitivity analysis for single parameters. This tool provides a rapid glimpse on the technology performance and supports the identification of the parameters that contribute the most to the overall uncertainty, ultimately allowing to make preliminary considerations about product development strategies. It is therefore a potentially useful, easy-to-use tool that can be used by manufacturers as a first exploratory analysis on cost-effectiveness.

However, some limitations of the MAFEIP tool are outlined. First the possibilities to structure the decision problem are somewhat limited. For example, the cycle length of the model is fixed to 1 year and there is no easy way to model shorter cycles, that may be required when modelling certain conditions. In addition, while the tool allows to dynamically adapt the probability of death to the aging of the cohort through official life tables, there is no way to make the model dynamic also for other parameters. For example, in this case study, it would have been relevant to allow for higher risks of fall-related injuries or death as the cohort ages through the model. Furthermore, since in the sensitivity analysis it is only possible to explore variations for single parameters at a time, problems may arise when the same parameters are used for both the treatment and control groups. In this case study, the relative risk of mortality when experiencing a fall was assumed to be the same in both groups, since the intervention is about preventing a fall, and does not affect the probability of injury once the fall has occurred. However, when conducting the sensitivity analysis users must choose whether to vary such parameter in either one or the other group, which makes the results of the analysis meaningless. In addition, correlation between parameters is also neglected. However, it often occurs that that some of the parameters used in the model are calculated from other parameters, and therefore, varying the former would require the latter to vary accordingly. For example, the incidence of falls in the intervention group was calculated using the baseline probability of falling, the probability of having OH and the data on the sensitivity of the device. Therefore, while it was possible to conduct sensitivity analysis on the overall incidence of falls with the intervention, sensitivity analysis on the baseline incidence of falls was not possible since it would have required both probabilities in the control and intervention groups to vary accordingly to the underlying formula linking them. A more flexible way of populating the model, allowing for example to use formulas to input model parameters, would highly increase the flexibility of the tool.

Lastly, incorporation of additional features, such as other standard and emerging methods for economic evaluations would highly increase the relevance and usability of MAFEIP. Particularly, especially for early stage interventions, the possibility of conducting probabilistic sensitivity analysis (PSA) is a must-have feature, as PSA allows the user to explicitly characterize the existing joint model uncertainty and to make statements in probabilistic terms, rather than just having a (considerably uncertain) point estimate on cost-effectiveness [28]. Also, Value of Information analysis may enrich the results provided by the MAFEIP tool. In short, VOI analysis can quantify the impact on the expected net monetary benefit of reducing the uncertainty over all or a subset of model parameters. In turn, VOI analysis could inform what further research would maximize the returns of R&D investments using the same decision rule used for assessing cost-effectiveness analysis [29,30,31]. VOI analyses has not seen a widespread diffusion so far, its use is rapidly expanding, also thanks to methodological innovations that strongly reduced its computational burden [30]. Use of VOI in early economic evaluations may provide additional relevant elements to inform product development and evidence generation plans. It may also provide a more explicit and transparent framework to conduct early negotiations on future research needs, price and adoption decisions. For example, VOI analyses may be used to agree on the manufacturer clinical (and economic) evidence generation plan during early dialogues (EDs) between manufacturers and national HTA bodies [32, 33], such as the ones coordinated by the European Network of Health Technology Assessment (EUnetHTA), which is now launching a new ED procedure specifically for medical devices (personal communication).

In addition, by identifying the parameters that contribute the most to decision uncertainty, VOI could support early negotiations over the possibility to define some form of performance-based risk sharing arrangements (PBRSAs) at market access. PBRSAs, are payment schemes where the amount or level of reimbursement is made conditional to the collection of further data and the confirmation of the expected performance of a technology in a defined patient population over a specified period of time [34, 35]. When considering such type of contractual agreements, VOI analyses may first inform whether further research is needed (thus supporting judgments on whether a PBRSAs would be recommended), and second outline which parameters in the model would need to be investigated further through additional data collection.

Limitations of this study are also reported. The presented case study was developed mainly to discuss the applicability of the MAFEIP tool rather than estimating the cost-effectiveness of fall-prediction devices. Therefore, parameters used in the model were not the results of a thorough systematic literature review, and some relevant studies may have been missed. In addition, in this simplified case study, no subgroup analyses were performed on different patients’ characteristics, for examples patients with multiple comorbidities or physical impairments.

7 Conclusions

The MAFEIP allows to quickly create early economic models, and to explore model uncertainty by performing deterministic sensitivity analysis for single parameters. However, the integration within the MAFEIP of common analytical tools such as probabilistic sensitivity analysis and Value of information would greatly contribute to its relevance for evaluating innovative technologies within and beside the EIP on AHA initiative.