FormalPara Key Points for Decision Makers

Value of Information (VOI) is powerful tool for understanding priorities for future research and ensuring value for money from proposed clinical trials.

There are differences between standard methods for designing clinical trials to understand the safety and efficacy of healthcare interventions and the VOI approach, which can lead to inappropriate research designs.

We demonstrate the critical importance of including all interventions that could be used for the health condition under investigation when using a VOI-based trial design.

1 Introduction

Value of Information (VOI) methods have long been touted as tools for research prioritisation [1,2,3,4] and trial design [5, 6], particularly based on exploring uncertainty in health economic decision models [7, 8]; however, recent efforts have highlighted their potential for clinical trial design [3, 9]. Practical applications of VOI are becoming more widespread, [10,11,12,13,14,15,16] thanks in part to novel methods to efficiently calculate VOI measures [17,18,19,20,21,22,23,24].

VOI methods can compute the economic benefit of reducing uncertainty for decision makers through a new research study [25, 26]. Research studies that provide the greatest benefit, among a set of proposed alternatives, should be prioritised [1]. Crucially, when combined with a health economic decision model, VOI methods compute the benefit of research in monetary units (e.g., Great Britain Pound [GBP], United States dollars [US$]) meaning that the net economic benefit can be computed by subtracting the trial costs from the value [27]. This allows for coherent research prioritisation and trial design that supports efficient spending of research budgets. It also ensures that research provides value for money when only trials with positive net value are funded [1]. Additionally, as research is valued in common monetary units across different disease areas, VOI can prioritise research across disease areas [25]. Finally, implementing VOI with health economic decision models ensures that research supports decision making about health policy within publicly funded healthcare systems [6, 28,29,30].

In contrast to VOI-based research design, clinical trials are usually designed in a multi-stage, multi-stakeholder process that aims to reflect differences in priorities and perspectives but does not provide a single coherent framework for research design across disease areas [3]. Research priorities for trials are selected through consultation with experts and stakeholders, often including policy/decision makers [3]. Research priorities can also be selected by determining areas with a high burden of disease or substantial variation in clinical practice. Once priorities have been identified, researchers and stakeholders will determine the comparators and outcomes for the proposed study. These may be informed by clinical interest and experience, the literature, and/or pilot work and feasibility. The sample size of the proposed trial is then selected to be sufficient to demonstrate statistical significance for a primary outcome of interest [31].

To expand the application of the coherent, VOI-based method for trial design, the key assumptions of the VOI framework must be understood by individuals who design trials [32]. Although many of these assumptions have been addressed in recent guidelines [25, 26], one key assumption was not highlighted—specifically, the assumption that correct VOI-based trial designs require that all relevant interventions for the disease of interest should be evaluated in the health economic decision model. This assumption contrasts with standard methods for trial design, which focus on the interventions that have been selected for inclusion in the proposed trial through external prioritisation processes. The issue of excluding interventions has been discussed when developing health economic decision models generally [33] and alongside clinical trials [34] or when evaluating screening interventions [35]; however, they have not been discussed for VOI analysis and are less familiar to individuals who design research.

Another critical assumption of VOI is that the information collected through research is added to the evidence base of the original decision model [25]. This is different from standard trial analysis, where the primary results from the trial are most often reported in isolation using statistical tests. Once these primary results are reported, they may be added to a meta-analysis, but it is rare to account for this future meta-analysis in the trial design [36, 37]. However, for VOI analyses, it is crucial to include all available evidence in the final decision to ensure meaningful treatment recommendations following trial completion. The importance of including all available evidence in a health economic model has been highlighted previously [38, 39] but its impact on research design using VOI has, to our knowledge, not been demonstrated.

This paper uses two hypothetical examples to highlight the importance of (1) evaluating all the interventions that might reasonably be considered for the patient population being studied, and (2) the need to retain the full decision model following the trial. These examples highlight that using VOI in research is an iterative process, and if the decision model excludes a treatment option at any point, then incorrect recommendations could be made in either the research prioritisation or future adoption of effective interventions [9].

2 Value of Information Analysis

All VOI analyses are based on a decision model, which aims to determine an optimal intervention from a set of alternatives. Typically, VOI has been implemented in practice using health economic decision models, which evaluate the costs and benefits of different health care interventions for a specific disease. These models synthesise the available evidence to calculate the interventions’ costs and benefits before combining these two measures into a summary of net monetary or health benefit to determine the optimal intervention [40]. To accurately determine the optimal intervention, the model should evaluate all relevant interventions for the disease using all available evidence [33, 41].

The evidence base for health economic decision models is normally informed by different sources, including previous clinical trials, meta-analyses, literature reviews, observational studies and clinical judgement [39, 42]. This evidence defines model parameters that are combined to compute the net benefit, often through a complex model structure that represents the disease course and how the interventions could improve prognosis or quality of life. As parameters are hardly ever known with certainty, their uncertainty is modelled using probability distributions [42, 43]. This uncertainty is then translated into uncertainty in the net benefit for each intervention. VOI analyses then estimate the economic value of reducing this parameter uncertainty. Thus, as highlighted previously, VOI can only accurately prioritise research if these probability distributions truly represent the uncertainty in the model inputs [25].

If all relevant interventions have been enumerated and parameter uncertainty modelled correctly, then based on the current information, a risk-neutral decision maker should implement the intervention with the highest expected net benefit [40]. However, as the parameters are not known with certainty, it is possible that the intervention with the highest expected net benefit is not the cost-effective intervention. For example, if the current evidence underestimates, on average, the effectiveness of a novel drug, then updated information could demonstrate that this novel drug is cost effective. Conversely, if the current evidence overestimates the effectiveness of the novel drug, then it could be deemed cost ineffective.

Collecting additional information can reduce the chance that an inefficient intervention is implemented, thereby removing the potential financial burden of implementing a non-optimal intervention. This burden, equal to the VOI, is calculated as the difference between the value of the current optimal intervention and the value of the intervention that is deemed optimal after the additional information has been collected [44]. This can also be thought of as the financial loss associated with implementing an inefficient intervention. Thus, VOI accounts for the chance that additional information would change the optimal intervention and the associated loss of the current decision in terms of wasted financial resources.

The VOI for a specific trial can be compared with the cost of running the trial [45]. If the study cost is higher than the VOI, the potential loss from implementing the current optimal intervention is lower than the cost of gathering additional information, and thus the trial is an inefficient use of resources [1, 45]. Among trials with net positive value, we can search for the design with the highest value, which would be the optimal study for reducing uncertainty in a specific disease area. Trials can then also be prioritised by value across clinical areas to determine the most efficient allocation of research funding. VOI can be used to design non-randomised studies but we focus on randomised clinical trials as these are more expensive and require a comprehensive design process.

3 The Importance of Including All Decision Options in Trial Design

Correctly specifying the uncertainty for the model inputs is crucial for VOI analysis [6, 16, 25, 32]. We now demonstrate that including all feasible interventions in the health economic decision model is also important. Crucially, this consideration is not common in trial design where methods only focus on the proposed interventions for the trial. For example, sample size calculations only specify the sample size for the selected trial interventions.

To illustrate the importance of including all interventions that are used within a jurisdiction and population of interest [41], we introduce a hypothetical example. For illustrative purposes, we have not developed a full health economic decision model but rather sampled the costs and effects from a multivariate normal distribution (Online Resource). However, to facilitate the discussion and explore how relevant interventions may be excluded from a health economic model, we contextualise these results.

We assume that a novel painkiller has been developed to treat chronic back pain, and a trial to compare the effectiveness of this novel treatment with the standard of care (SoC), self-medication with over-the-counter painkillers, has been suggested. Figure 1 displays the differences in the simulated costs and effects for a comparison between the novel painkiller and the SoC. The novel painkiller is, on average, £515 more expensive but also more effective, with a mean gain in effectiveness of 0.27, as indicated by the x and y coordinates of the red cross. For a willingness to pay of £25,000 (represented by the line bisecting the plane), the probability that the novel painkiller is cost effective is 0.61 (the proportion of points in the grey area).

Fig. 1
figure 1

The incremental costs and effectiveness for a health economic model comparing the novel painkiller with the current SoC. Points in the shaded area indicate that the novel painkiller is cost effective and points elsewhere indicate that the SoC is cost effective. The x-axis shows the effectiveness differential in terms of QALYs, while the y-axis shows the difference in population costs. The ICER is shown as a red cross. SoC standard of care, ICER incremental cost-effectiveness ratio, QALYs quality-adjusted life-years

Full size image

A VOI analysis, using the Expected Value of Perfect Information (EVPI) [44] to compute the value of resolving all parameter uncertainty, results in £7063 for each patient who experiences chronic back pain (Online Resource). The annual prevalence of chronic back pain is between 650,000 and 1 million [46], therefore if all these 1 million individuals could receive this novel painkiller to treat their back pain, the upper limit for the monetarised benefit of the proposed trial is around £7 billionFootnote 1. Therefore, this VOI analysis indicates that there is likely to be value in a future trial, although further VOI analyses using a complete health economic model would be required to compute the value of a trial [27].

However, by focusing on the interventions for our proposed trial, we unintentionally exclude alternative interventions to treat chronic back pain—for example, physiotherapy. To address this, we extend our hypothetical example to include a third ‘physiotherapy’ option, with Fig. 2 displaying the incremental costs and effects for physiotherapy against the SoC (black dots) for the painkiller against SoC (grey dots). We assumed that physiotherapy is, on average, £7160 cheaper than the SoC and leads to a mean increase of 2.98 QALYs, making it more effective and cheaper than the novel painkiller as well.

Fig. 2
figure 2

The incremental costs and effectiveness measures for a full health economic model evaluating the novel painkiller (grey dots) and physiotherapy (black dots) for the treatment of chronic back pain. The black dots are almost all in the shaded area, indicating that physiotherapy is cost effective compared with the SoC. The black dots are lower and to the right of the grey dots, indicating the physiotherapy is cost effective compared with the novel painkiller. SoC standard of care

Full size image

The EVPI for this augmented model, which includes interventions beyond those in the trial, is £0. Thus, the proposed trial would waste research resources and should not go ahead. While this example is hypothetical, it demonstrates that VOI-based research recommendations could be incorrect if the full set of interventions is not considered. This is true even when considering trials that do not evaluate all interventions. In practice, there are many ways that models could exclude interventions [47]; for example, the investigators only consider drug interventions or are planning a placebo-controlled trial. Note also that including all relevant interventions could also increase decision uncertainty and inflate the VOI, leading to trials with higher value. For example, if we include an option where individuals receive both physiotherapy and the novel painkiller, the VOI may increase as this option would have uncertain efficacy that exceeds the efficacy of physiotherapy alone.

4 Trial Analysis with a Value of Information-Based Design

Next, we highlight the importance of retaining the full decision model once data have been collected by considering a hypothetical example that assumes a novel drug to treat depression has been developed. Similar to before, this illustrative example only samples the costs and effects from a multivariate normal distribution (Online Resource), which means that it does not accurately assess treatments for depression, and key targets for future research cannot be identified. We can however explore how including different interventions in the VOI calculations affects the value of research.

We assume that evidence for the novel depression drug is limited as it is supported by a single trial with limited recruitment. As there are both pharmaceutical and psychological interventions for depression, we include four potential interventions for mild-to-moderate depression: cognitive behavioural therapy (CBT), exercise programmes, standard drug-based therapies (SoC) and the novel drug. Figure 3 displays the incremental costs and effectiveness between the SoC and either the novel drug (black dots), exercise (blue triangles) or CBT (grey crosses). We generated the costs and effectiveness for the novel drug with significantly larger uncertainty, compared with exercise and CBT, to mimic the idea that limited data are available for the novel drug. Table 1 highlights that CBT has the highest average effectiveness and the lowest average cost, given the available evidence, while exercise is less effective and more costly than the SoC. The novel drug is cheaper and more effective than the SoC, on average, with significant uncertainty.

Fig. 3
figure 3

Cost-effectiveness plane for the health economic analysis comparing a novel drug (black dots), exercise (blue triangles) and CBT (grey crosses) against the SoC for mild-to-moderate depression. This model is illustrative and does not evaluate treatments for depression. SoC standard of care, CBT cognitive behavioural therapy

Full size image
Table 1 Expected costs and effectiveness measures for our illustrative health economic model comparing treatments for mild-to-moderate depression
Full size table

To design a trial using VOI, we evaluate all possible combinations of treatments that could be included in our trial (Table 2). We calculate this by assuming that perfect information is obtained for the interventions in the trial and no information is collected about the interventions excluded from the trial (Online Resource). Trials that investigate the novel drug are valuable, with a minimum value of £2783 per person. The highest value is associated with trials that investigate CBT alongside the novel drug. This is unsurprising as these are the two interventions with the greatest potential to be cost effective following the trial completion.

Table 2 The maximum value that could be obtained by performing further research to determine the cost effectiveness of the different treatments for mild-to-moderate depression
Full size table

However, a trial that randomises between the novel drug and SoC has a similar maximum value, with a difference of £1. If including CBT in the clinical trial had significantly greater trial costs than a trial that randomises between two drugs, the optimal trial, accounting for costs, may exclude CBT. However, formally assessing this would require an in-depth VOI analysis using the Expected Net Benefit of Sampling, which should also consider the cost of randomising to non-optimal therapies [5].

Thus, a comprehensive VOI analysis could exclude the most effective intervention from the proposed trial. This highlights the importance of not only considering all interventions when designing the trial but retaining all interventions in the model upon trial completion to ensure that the correct optimal intervention is selected. If analysis was restricted to the interventions included in the trial, then we could make incorrect policy recommendations. While this advice chimes with standard advice for health economic decision modelling [34], it is counterintuitive for researchers who specialise in adaptive trial design, where interventions that are dropped from a trial are excluded from the final analysis [48, 49].

4.1 Ethical Considerations for Value of Information-Based Designs

There may be ethical issues with trials that exclude the most effective intervention, given current information, as trial participants cannot receive the optimal (i.e., cost effective) intervention. However, by focusing on the novel drug, we reduce decision uncertainty faster, with lower costs and fewer trial participants. Thus, the cost-effective intervention can reach patients faster and clinical trials will stop earlier. From a social perspective, this reduces the total opportunity cost of decision making for both research and treatment recommendations, thereby maximising the health of the population.

Furthermore, our information about CBT will have been generated by many previous trial participants. Thus, VOI determines that there is limited additional value in studies that focus on CBT; sufficient evidence is available to conclude about its cost-effectiveness. In this sense, VOI prioritises additional research that will redress the imbalance in the number of participants receiving each treatment. Finally, VOI is based on cost effectiveness, rather than clinical effectiveness. As the most clinically effective intervention may not be the cost-effective intervention, participants in a trial that excludes the cost-effective intervention could receive a more effective intervention. In this setting, trial participants receive a superior treatment by retaining cost-ineffective treatments in the proposed trial.

These ethical considerations presuppose that the VOI analysis has been carried out correctly; the economic model compares all relevant interventions, the costs and benefits of these interventions have been accurately assessed, and uncertainty in the model inputs has been correctly identified. Therefore, these considerations are critical to ensure benefit from clinical research using VOI designs.

5 Conclusions

VOI offers a principled method for research prioritisation and trial design based on the net economic benefit of reducing decision uncertainty. As VOI measures aim to reduce decision uncertainty, rather than demonstrate clinical effectiveness, they can provide alternative study designs to traditional methods. VOI can ensure that cost-effective interventions reach patients faster by guaranteeing that research supports policy making within publicly funded healthcare systems. However, incorrect specification of a health economic model, ignoring potential interventions or incorrectly characterising uncertainty, can lead to misleading VOI results and waste research resources. Additionally, if the trial is not analysed within the wider evidence base of the original health economic model, then ineffective interventions could be widely implemented.

VOI analyses can lead to alternative research designs compared with standard methods that take a clinical perspective. However, they also ensure that policy decisions are formally considered in the conceptualisation and design of clinical research. This has huge potential to increase the relevance of research beyond the clinical question under consideration [37, 50].