FormalPara Key Points or Decision Makers

Consideration of risk assessments is currently not common practice in health technology reimbursement decision-making, hampering appropriate risk management.

Risk communication is challenging and particularly so because risk assessments often do not take account of all uncertainties. There is no clear link between risk assessments and appropriate risk management strategies.

Two new risk communication tools, the Assessment of Risk Table (ART) and the Appraisal of Risk Chart (ARCH), may help assess risk, facilitate communication between analysts and policymakers, and guide the appropriate use of available risk management strategies.

1 Introduction

The results of health technology assessments (HTA) are uncertain, and decision-making is associated with a risk, which is rarely incorporated into deliberations about health technology reimbursement. This risk has two components—there is a probability of making the “wrong” decision and this comes at a cost [1,2,3]—and both elements are subject to uncertainty. In a healthcare system that aims at maximizing efficiency, this “wrong” decision is defined as recommending a new technology for reimbursement in the healthcare system when, in fact, it is not the cost-effective option, or vice versa. In a world of perfect information, the probability of a rational policymaker reaching the “wrong” decision would be nought. In healthcare, and increasingly given recent developments in pharmaceutical regulation, a state of perfect information is not attainable, and healthcare reimbursement authorities must deal with more uncertainty than ever before. Health technology reimbursement decision-makers are accustomed to considering uncertainties in the form of deterministic sensitivity analyses, scenario analyses, and probabilistic analyses (PA) [4, 5], although the latter are not a standard requirement in all settings. Risk, however, is rarely assessed, which may hamper transparent and consistent risk management.

Many jurisdictions make available risk management strategies that can be employed in situations where risk is considered large [6]. These are called managed entry agreement (MEA) schemes or (performance-based) risk sharing arrangements [1, 7], which are agreements between healthcare systems and manufacturers that facilitate patient access to new technologies whilst managing risk for the healthcare system. Although terminology differs, existing publications agree on these MEAs broadly comprising two types of schemes: financial and data collection schemes [6, 8, 9], which can assume a variety of different possible specifications and combinations. The ISPOR-SMDM taskforce [7] and other publications on risk assessment frameworks [1, 8, 10] stipulate that a value of information (VOI) framework be used to assess the costs and benefits of different MEAs. This entails calculation of the expected value of perfect information (EVPI) as the overall risk metric, as well as the expected value of sample information (EVSI) to assess the benefit of reducing uncertainty with a particular research study design [11, 12]. In practice, such evaluations are rarely performed [13], and reviews of data collection-based MEAs have shown that uncertainties in an assessment are not systematically considered and VOI is not routinely performed when devising MEA schemes [14, 15].

Risk communication is, to our knowledge, currently neither standardized nor a routine consideration in HTA. The absence of quantified risk estimates through VOI may be one barrier; another is that whilst risk can be quantified through VOI, many uncertainties are typically not parameterized and therefore not included in the quantitative risk estimate. Uncertainty taxonomies distinguish between statistical uncertainty, scenario uncertainty, recognized ignorance, and total ignorance [16]. These categories relate closely to the spectrum of the known known (determinism) via the known unknown all the way to the unknown unknown (ignorance). More recently, a taxonomy distinguished between imprecision (which relates to statistical uncertainty), indirectness or bias (scenario uncertainty), and unavailability (recognized ignorance) in the field of health economic evaluation [17]. Statistical uncertainty or imprecision can be incorporated in models with relative ease. It is more concerning that scenario uncertainty, or indirectness, and recognized ignorance, or unavailability, are often not parameterized. Structural uncertainty is one example of uncertainty that is typically not parameterized [16, 18]. Risk communication would therefore have to integrate uncertainty that is parameterized and that which is not parameterized.

When there is large risk, reimbursement authorities generally have the following decision options [7,8,9,10]: negative or positive reimbursement decisions, and MEAs. However, the link between assessment outcomes and appropriate and available risk management options may be unclear to health technology reimbursement decision-makers. When this link is unclear, policymakers may perceive risk as a threat, instead of using the opportunity to manage it. This reaction has been described in the literature as uncertainty intolerance, that is, negative beliefs about uncertainty that may result in negative reactions towards it [19, 20]. Policymakers may also react to the public’s perceived uncertainty intolerance [21]: if people in charge admit to uncertainty, does this not undermine the public’s trust? Recent research has shown that this is not necessarily the case, and the key is in how uncertainty is communicated, but this remains challenging and research on developing communication tools is sparse [22]. We consider that there may be a need for risk communication tools in HTA.

The aim of this study was to develop risk communication tools for use in HTA by health technology reimbursement decision-makers. It is expected that these decision-makers will have literacy in health economic evaluation methods. We focus on two features in particular: [1] inclusion of quantified and non-quantified risk assessments in risk communication and [2] linking risk and net benefit outcomes to appropriate risk management strategies.

2 Methods

2.1 Scoping Review of Risk Communication Tools in Health Technology Assessments (HTA)

We performed a scoping review with the aim of identifying tools for uncertainty and risk communication in HTA. We included uncertainty in the review to enable us to spot gaps between uncertainty and risk communication. Our citation pearl-growing search started with a publication on uncertainty assessment—the ISPOR-SMDM task force publication Briggs et al. [23]—and a publication on assessment of MEA schemes—Garrison et al. [7]. Citation pearl-growing is a search and review of references citing the selected publication(s) and their reference lists [24]. The rationale for choosing these two articles was that uncertainty assessment tools in HTA are likely to refer to the ISPOR-SMDM taskforce paper and that consideration of risk is arguably necessary to inform decisions on MEA schemes. We excluded publications that did not describe a method for communicating uncertainties and/or risk and publications that were not within the topic of health technology decision-making. The identified tools are briefly presented, along with a verdict on whether they met the following criteria: inclusion of presentation of quantified and non-quantified risk assessment and inclusion of a link between risk assessment and appropriate risk management strategies.

2.2 Development of Risk Communication Tools

We used publications on communication of risk and uncertainty by van der Bles et al. [22] and Fischhoff et al. [25] as inspiration for developing risk and uncertainty communication tools. Factors to consider in communicating uncertainty according to van der Bles et al. include who communicates, what is being communicated, and in what format, to whom and to what effect [22]. In the development of risk and uncertainty communication tools, according to Fischhoff et al., consideration must be given to [25] the goal of uncertainty and risk communication (change of beliefs versus change in behaviour); how to evaluate the tool (through usability testing, a process evaluation, and/or an outcome evaluation); and tests for adequacy of communication (whether it contains the information that users need, makes information accessible, and can be understood). Based on these considerations, we drew up our development framework (Table 1).

Table 1 Development framework for HTA risk assessment and communication tools
Full size table

Because our desired effect was twofold (changes in knowledge and behaviour), our object of communication was twofold, and we developed two tools. We developed the Assessment of Risk Table (ART) to summarize cost-effectiveness, uncertainty, and risk information in a one-table overview. For this, we required information on standard reporting of health economic submissions, and we used the TRansparent Uncertainty aSsessment Tool (TRUST) [17] as well as health economic submission guidelines and report templates from the Dutch Health Care Institute and England and Wales National Institute for Health Care and Excellence (NICE) as starting points.

For our second objective, linking risk outcomes to appropriate risk management strategies, we developed the Appraisal of Risk Chart (ARCH). To do so, we drew on the literature about relevant risk management strategies and decision frameworks [6,7,8,9,10].

2.3 Evaluation of Risk Communication Tools

To test the adequacy of the information presented in both tools and their user-friendliness, we performed individual in-depth semi-structured interviews with international stakeholders (N = 11). These stakeholders were chosen to represent the following stakeholder groups, with the purpose to include a variety of different end users of these tools, that is, analysts, reviewers, and decision-makers: HTA policy advisors from the Dutch Health Care Institute (n = 2), independent academic advisors for HTAs (n = 4), two of whom were part of the Dutch pharmaceutical decision-making committee (Wetenschappelijke Adviesraad), an appraisal committee member (n = 1) for the England and Wales NICE, and industry stakeholders (n = 4, one working in a pharmaceutical company and three for HTA consultancies). Convenience sampling was used to establish contact. The interviews were recorded, and statements relevant to the design, usefulness, information adequacy, and potential use of the tools were transcribed by two researchers. Member checks were conducted, and feedback was incorporated into final interview records.

For illustration purposes, we applied the ART and ARCH in the assessment of pharmacological treatments for peripheral artery disease [26], for which all required data were publicly available. For further evaluation, we applied the ART and ARCH in a real-world prospective case, an assessment of tomosynthesis versus digital mammography for use in population breast cancer screening [27]. We recorded notes of our observations and meeting minutes. A workshop was held with stakeholders (N = 15) from the Dutch HTA setting to test the tools in potential end users, including decision-makers (n = 8), some of whom were healthcare providers, from the pharmaceutical industry (n = 1), and representatives of several academic institutions (n = 6) involved in the assessment of new health technologies. Workshop materials and results were collated and translated into meeting notes, which were circulated with all participants for member checks.

Ethical approval was not required for this research as it did not involve any interventions in human subjects and as such was not subject to the Medical Research Involving Human Subjects Act (Wet maatschappelijke ondersteuning [WMO]) [28]. We followed the “Netherlands Code of Conduct for Research Integrity” [29].

3 Results

3.1 Review of Risk Communication Tools

The pearl-growing citation search resulted in 664 citations of the two starting-point publications in Web of Science (17 September 2020), and after removal of duplicates, 641 remained. After title and abstract screening, 25 publications were included in the full-text review. After further exclusion, four relevant publications were identified [1, 17, 30, 31]. From searching references of the included studies, a further six relevant publications were identified [32,33,34,35,36,37]. Together with the starting-point publication Briggs et al. [23], which included a relevant tool, this resulted in eight included tools (Table 2) [1, 17, 23, 30,31,32,33,34,35,36,37].

Table 2 Review results: Uncertainty and risk communication tools
Full size table

Only a few tools for uncertainty and risk communication were identified. Three of these facilitate risk assessment in HTA: the EVPI curve, expected loss curves (ELCs), and the HTA risk analysis chart with the payer strategy and uncertainty burden (P-SUB). None of these take non-quantified uncertainty into account and all are solely based on PA. None of these clearly link assessment outcomes to appropriate risk management strategies, such as research and/or financial schemes.

3.2 Proposed New Risk Communication Tools

3.2.1 The Assessment of Risk Table (ART)

The ART presents outcomes from the HTA such as cost-effectiveness and incremental net benefit together with descriptive information about uncertainty and quantitative risk estimates. The stylized version is presented in Table 3. The full version of the ART and ARCH can be found in Appendix 1 (see the electronic supplementary material) or following the link https://osf.io/v2a9r/. In addition, an illustrative example is presented in Appendix 3 (see the electronic supplementary material). Part 1 includes general information on the threshold, the disease burden, and the population size. Part 2 includes the most plausible expected incremental cost-effectiveness ratio (ICER) with estimates of its upper and lower end, and the probability of cost-effectiveness at the given threshold. Part 3 presents risk, or the EVPI, and expected incremental net benefit/loss estimates. Incremental net benefit/loss was used to overcome well-known shortcomings of the ICER [38]. In addition, net benefit is considered appropriate for estimating opportunity losses associated with a sub-optimal decision—it represents the benefit forgone by not choosing the optimal option [1, 9, 30, 33,34,35]. The user can express risk and net benefit in monetary or quality-adjusted life-year (QALY) terms, at the per person or population level, as appropriate in their decision-making context. Part 4 enables an overview of uncertainties with the highest impact on outcomes. It separates those uncertainties that are modelled probabilistically, and for which an expected value of perfect parameter information (EVPPI) can be estimated, from those uncertainties that are not reflected in the PA. If uncertainties are inserted in this latter category, the EVPI cell turns red (full version, Appendix 1 in the electronic supplementary material); this implies that EVPI does not equal actual risk because important uncertainties are not parameterized. The impact of uncertainties on outcomes can be determined with EVPPI analysis, scenario analysis, and the judgement of stakeholders. For each uncertainty, information can be entered as to whether the identified uncertainty could potentially be addressed with feasible research designs.

Table 3 The Assessment of Risk Table (ART)
Full size table

Table 4 shows the ART filled in for the illustrative case study of pharmacological treatments for peripheral artery disease [26]. Dual platelet therapy (DPI) with rivaroxaban plus aspirin was compared with single platelet therapies—clopidogrel and aspirin. Clopidogrel was dominated by the other two strategies, and the ART therefore shows the comparison of DPI versus the best comparator, which was aspirin. At the relevant threshold, DPI was expected to be cost-effective when compared to aspirin. Based on cost and effect information, the expected incremental net benefit was €2.1 billion at the population level per year. Risk, as quantified by the EVPI, was €535 million at the population level per year. The most important uncertainties that were parameterized and therefore in the PA were utilities and costs. There were important uncertainties that were not parameterized, including uncertainty about the model structure, the relative effectiveness of treatments, and the generalizability of utility values. To include all uncertainties in the model, the authors developed an “integrated model” in which these uncertainties were parameterized as much as possible; this led to minor changes in all reported values (results shown in Appendix 3 in Supplementary material).

Table 4 ART for pharmacological treatments for peripheral artery disease in the Netherlands
Full size table
3.2.1.1 Considerations in the Development of the ART

In part 1, disease burden in the current appraisal was added based on interviewees’ comments; this is likely to be especially relevant in the Dutch setting and can be removed if irrelevant. In part 2, participants requested the addition of fields for reflecting plausible cost-effectiveness ranges, which we incorporated as fields for the upper and lower bound of the ICER. Other participants considered these difficult to use as they were unsure how estimates would be obtained. In response, we added comment fields (extended version) so that detail on the methods used for estimation can be provided. Useful ranges likely differ for assessment groups, committees, and case by case (for instance, extreme bounds versus plausible range). Presentation of the company’s versus the review group’s preferred ICERs could be considered, but we suggest the reporting of a consensus most plausible ICER or range (other ICERs can be added manually if so desired).

In part 3, we initially presented the sum of expected net benefit/loss and EVPI for a full risk assessment, as is done elsewhere [1, 30, 35]. However, based on interviews and our own reflections, we finally opted for separate presentation as we consider these two measures as dimensions of a decision problem that may best be viewed separately to facilitate the choice of risk management strategy (see presentation in the ARCH). Interviewees agreed with this assessment and considered net benefit and EVPI to be appropriate metrics for risk management considerations. In part 4, we opted against including all identified uncertainties, to avoid information overload and because we consider that such a summary should be provided prior to filling in the ART.

Stakeholders in this exercise found population values of EVPI to be more suitable for decision-making than per person estimates. To enable assessment of whether population size or decision uncertainty was the main driver of a risk assessment, we enabled the inclusion of both in the extended version. In addition, the EVPI measure was criticized for not being informative without a reference value for an acceptable magnitude. Some stakeholders voiced the concern that reporting quantitative risk assessments in the ART may shift the focus right back to parameterized uncertainty and may tempt the audience to ignore other important uncertainties. In response, we adapted the presentation of non-quantified uncertainties in the ART by adding conditional formatting, where EVPI cells turn red when information is entered in the “Uncertainties not in PA” cells.

3.2.2 The Appraisal of Risk Chart (ARCH)

The ARCH links back to the outcomes of the ART and presents combinations of expected incremental net benefit (a measure of cost-effectiveness) and risk outcomes with possible policy options. The ARCH is a diagram with two axes (Fig. 1; editable versions of both the ART and ARCH can be found in Appendix 2 in the electronic supplementary material or https://osf.io/9zfqe/; an illustrative example is in Appendix 3). The x-axis presents the magnitude of risk on a scale from low to high; the y-axis presents the expected incremental net loss or net benefit of the technology of interest over the next best alternative in terms of its cost-effectiveness—both derived from the ART. The four resulting quadrants of combinations between risk and expected net benefit call for distinct policy options. The proposed policy options are simplified to their most abstract levels: positive recommendation, financial MEA (for example, a price scheme), recommendation with research (RwR), only in research (OiR), and negative recommendation. Details on possible scheme designs within these categories are available elsewhere [7,8,9]. Risk on the x-axis is not based solely on the EVPI as a risk measure, but can, in addition, be informed by judgement (when not all uncertainties are parameterized). The assessment of whether there is high or low risk can be informed based on the ART. The rationale for risk on this axis is that it can help decide whether an MEA may be of value. The expected net benefit/loss on the y-axis is based on simple expected incremental net benefit calculations of the technology of interest versus its “best” comparator. The rationale for using this metric was that it avoids the ambiguity of the ICER [38] and drives the decision on a positive versus negative recommendation [10].

Fig. 1
figure 1

The Appraisal of Risk Chart. CED coverage with evidence development, MEA managed entry agreement

Full size image

The ARCH shows four quadrants, and a rectangle can be placed onto the ARCH to indicate the position of an assessment. Quadrant I represents low risk and positive incremental net benefit. Here, a positive reimbursement recommendation may be warranted. In quadrant II, a high risk and positive incremental net benefit implies that RwR may be indicated, and that a financial scheme may aid in reducing the risk to the healthcare system. Quadrant III represents low risk and an incremental net loss, that is, the new technology is expected to be less cost-effective than at least one of its comparators. Here, a negative recommendation may be indicated. Alternatively, a financial MEA could move the assessment up on the y-axis to quadrant I and may be worth exploring. Quadrant IV entails high risk and an incremental net loss. Here, a negative recommendation could be replaced by a financial MEA, which may change the position of the rectangle to quadrant II (to being expected to be cost-effective). Alternatively, an OiR strategy, or its combination with a financial scheme, may be worth exploring. One consideration is that when price changes, the position of an assessment in the ARCH may change entirely: with a reduction in price, a low risk may change into a high risk, for example, from quadrant III to II or to IV if a previously clearly not cost-effective technology now has an ICER close to the threshold. It is important to note that in the event of any changes, such as changes in the effective price or the evidence, the model must be re-analysed and the ART and ARCH reconsidered.

Figure 1 is filled in for the case of pharmacological treatments for peripheral artery disease. The integrated model from the case study was used (Appendix 3 in Supplementary material) as all potentially impactful uncertainties were parameterized and the risk assessment was therefore likely a good reflection of the actual risk. If the original model had been used, further thought would need to be devoted to whether those uncertainties not in the PSA would potentially drive up risk. Based on the outcomes detailed in the ART, DPI was expected to be cost-effective, and the rectangle was placed in the net benefit area of the figure, rather than the net loss area. Whether risk is considered high is for the decision-maker to debate; based on the numbers, we considered it moderate to high and placed the rectangle towards the high risk/net benefit quadrant. A recommendation with further data collection and/or a financial scheme may therefore be relevant options. The ART gives us a hint as to potential research targets.

3.2.2.1 Considerations in the Development of the ARCH

Even though the ARCH used dichotomizations (high and low risk, net loss, and benefit), we attempted to make borders look fluid. We recognize that there are no clear borders, first, because of uncertainty surrounding risk and net loss or benefit and, second, because there is no evidence for a threshold at which risk is considered high or low. There were conflicting views on this presentation; some interviewees preferred the fluid presentation and others preferred clear quadrants, possibly separated by the axes. We added instructions on how to position a particular assessment in the ARCH: [1] the rectangle can be placed in the quadrant that most characterizes the assessment as per stakeholder consensus; (2) the height and width of the rectangle can be adapted to indicate that the exact position is unclear.

Options without risk management were called “positive recommendation” or “negative recommendation”, terms that tested well in interviews and workshop. The distinction between RwR and OiR was considered useful as these policy schemes fall into different quadrants. Alternative terms are provided in brackets: “Data collection MEA/CED scheme” (where CED is coverage with evidence development) for RwR schemes and “Data collection MEA” for OiR. These alternative terms were considered important as some respondents were more familiar with some terms than others. We used “Financial MEA (Price Scheme)” to include both simple price discounts and more complex schemes (e.g. money-back guarantees), all of which we consider to be reductions in effective price. One comment was that “negative recommendation” in the net loss/low risk quadrant may not be appropriate where the population size is small. This is a budget impact consideration that does not match with the health economic/decision scientific framework currently in use in many jurisdictions. If this thinking were to be incorporated within our framework, this could be realized through a threshold that is adapted based on population size, to reflect that we are willing to pay more per QALY when the burden for the overall health system is relatively small. This may then change the assessment of uncertainties in the ART and result in a different position in the ARCH.

The interviews highlighted a lack of clarity on MEAs and policy options in local policy-making settings: it was unclear whether research recommendations and financial schemes would fall within the decision-making power of committees. We omitted the addition of “optimized use” policy options, which can be used to limit a recommendation to a specific subgroup, for instance, as recommendations for use in a subgroup require a new assessment (or subgroup analysis) and therefore an updated assessment of uncertainty. As users of the ARCH may miss this option, a disclaimer was added to the ARCH worksheet, suggesting that the ART and ARCH be reconsidered for the different subgroup. Re-assessment was not explicitly included as a distinct policy option in the ARCH; re-assessment is implicitly indicated as it should be part of any risk management scheme.

An overall concern was that speed of access may be reduced when there is a stronger focus on uncertainty assessment and risk management, which may delay HTA processes. In addition, proposing price schemes may be difficult in settings where price negotiations and HTA are performed in separate institutions (for example, in the Dutch context) and technologies are not re-appraised after the price negotiation has taken place.

4 Discussion

We developed two tools for communication of risk assessments: the ART and ARCH. The benefit of the ART is that it allows for comprehensive risk assessment based on all uncertainties—those included and excluded from quantitative risk estimates. The explicit consideration of all uncertainties in the same place as assessment outcomes is relevant since previous studies suggest that uncertainties are rarely considered when MEA schemes are designed [14, 15]. The ARCH provides policymakers with an overview of decision options including MEA schemes appropriate for a specific combination of incremental net loss/benefit and risk. Whilst these quantities have been used in uncertainty and risk assessment before [1, 30, 33,34,35] and the use of policy options is detailed elsewhere [7,8,9], the explicit link between combinations of risk and net benefit metrics to appropriate policy options in a simple communication tool is new. We consider that this may help policymakers in obtaining a simple overview of risk management strategies appropriate for each specific HTA. As such, the ART and ARCH may facilitate communication between analysts and policymakers, aid decision-making, and guide the appropriate use of available risk management strategies.

Our new tools address a clear gap in the literature as evidenced by our review: they provide the missing link between communication tools for comprehensive consideration of uncertainty and consideration of risk management strategies. Interviews and workshop revealed a general appreciation of the ART, with one participant stating, “This table goes a step beyond what we are currently doing.” The tools were carefully designed based on guidance on development of uncertainty and risk communication tools [22, 25] and evaluated in interviews as well as by use in a real-world HTA and concluding workshop. However, we consider that further application in real-world assessments in different settings may result in further insights, refinement, and/or adaptations. Given that we developed these tools from the Dutch and English perspectives, it is possible that the tools are not generalizable to all jurisdictions, and we invite interested parties to adapt them for their purposes, if necessary. An example could be the importance of disease burden, which determines the threshold in some jurisdictions. A disclaimer applies: the ART and ARCH are most appropriate in a decision-analytic context in which the aim is to maximize efficiency, that is, cost-effectiveness. Where decision-making is largely not based on cost-effectiveness but other considerations, the use of these tools may not be appropriate. It is a further limitation of this study that the use and usefulness of these tools were not prospectively evaluated in a pharmaceutical case. We did illustrate how these tools could potentially work in pharmaceuticals using a case study, but we acknowledge that their use in actual decision-making may lead to further adjustments. For example, even though the ART was designed with difficult-to-quantify uncertainty in mind, we cannot be sure that the ART and ARCH would perform well in the context of deep uncertainty, defined as “The condition in which analysts do not know, or the parties to a decision cannot agree upon (1) the appropriate models to describe interactions among a system’s variables, (2) the probability distributions to represent uncertainty about key parameters in the models, and/or (3) how to value the desirability of alternative outcomes” [39]. However, we think that our tools are the first that do attempt to include non-parameterized uncertainty and therefore go a step beyond what is currently done to include deep uncertainty. We also plan to update the tools when further insights become available.

A challenge for the uptake of these tools will be the requirement for VOI analysis. Currently, VOI analysis is not required by the majority of HTA bodies, and even where EVPI estimates are a requirement, these are often not considered further. We carefully assessed existing measures of uncertainty and risk in the context of HTA, and only VOI fulfils the requirements of a risk assessment: it combines the probability of an outcome with the value of the outcome. In line with the ISPOR-SMDM taskforce [7], we therefore strongly advocate for the use of VOI metrics in risk assessment and management.

We did not include budget impact considerations as this does not match with the health economic/decision scientific framework currently in use in many jurisdictions, which requires cost-effectiveness estimates at a given threshold. We concede that the same threshold may not be appropriate for technologies with a non-marginal budget impact. Such affordability considerations go beyond our framework in the presence of those technologies. Some jurisdictions such as the Netherlands include an a priori assessment of budget impact, which determines whether the technology is classed as low risk and reimbursed without further assessment of cost-effectiveness. This is to be viewed separately from the risk considerations that are considered here.

We consider that it may be necessary for the use of the ART and ARCH to provide training for decision-makers and analysts. This training should address background on methods used (VOI and net benefit) and on available policy options and relevant terminology.

The ARCH presents a simplification of reality and should not be understood as making policymakers’ deliberation obsolete—ideally, it should serve as a tool to support deliberation. The policy options in quadrants are only an indication of the most appropriate recommendations to enable an easy-to-grasp visualization, and we do not intend to be prescriptive. The dichotomization of risk into low and high is one such simplification—in reality, this is a gradient, and the assessment may be somewhat subjective. In addition, policy options may not be situated exclusively within these quadrants; there are instances in which a low risk could be addressed with “cheap” research or an effective lowering of the cost to the healthcare system. Further research should be performed on levels of risk acceptable to a healthcare system.

We wish to urge the reader not to understand the ART and ARCH as absolving analysts from providing uncertainty analysis that is as comprehensive as possible. The ART enables the presentation of uncertainties that are not parameterized and elevates them from unknown unknowns to known unknowns. But risk assessment will be more challenging the more uncertainty is not parameterized. Methods for quantifying uncertainty, including structural uncertainty, therefore remain crucial [18].

Where uncertainties remain excluded from the PA, both quantitative risk estimates, and subjective judgements need to be considered, and this could be challenging for decision-makers. It is therefore desirable that iterative processes enable the model to be adapted and all uncertainties to be incorporated in it, to quantify risk. However, this may not be feasible due to time and resource constraints and is most likely not possible in current HTA processes. Further research should focus on the integration of risk management in HTA processes.

5 Conclusion

In conclusion, we developed two tools for communication of risk assessments. We hope that these tools will help assess risk, facilitate communication between analysts and health technology reimbursement decision-makers, aid decision-making, and guide the appropriate use of available risk management strategies.