1 Introduction

Health economic evaluation (HEE) can support reimbursement decision making by comparing the cost and effects of health technologies. These evaluations are often based on decision-analytic models, and the results of these models always contain uncertainty [1]. Consequently, reimbursement decisions are often uncertain and are potentially associated with a risk of suboptimal decisions [2]. There is pressure on reimbursement institutions to grant market access at an early stage when evidence is immature [3]. As a result, decision makers are forced to make decisions given considerable uncertainty [4], which increases the potential benefit of assessing and managing uncertainty and risks adequately. Considering uncertainty in decision-making processes is challenging: its quantification and communication can be difficult [5] and it is sometimes met with uncertainty intolerance [6]. However, when uncertainties are quantified and communicated, they can be considered more easily and precisely, leading to better decisions.

Uncertainty assessment is considered good practice in HEE [1, 7, 8]. Uncertainty assessments can indicate the robustness of the results to changes in model inputs, the consequences of uncertainty associated with a decision, and the value of collecting additional information to support that decision. The latter can be calculated through value of information (VOI) analyses [9, 10]. The results of uncertainty assessments can inform the design of managed entry agreements (in the form of pricing schemes and/or evidence collection schemes) that minimise the opportunity loss associated with a reimbursement decision [11]. While uncertainty assessment is thus clearly relevant, it has been recognised that, in particular, uncertainty relating to structural aspects of the model is often not included in uncertainty analysis [12].

There is a lack of consistency in terminology around uncertainty assessment. The ISPOR-SMDM taskforce 7 paper on uncertainty assessment and parameter estimation [1] classified uncertainty into four types: stochastic uncertainty, parameter uncertainty, heterogeneity, and structural uncertainty. The TRansparent Uncertainty Assessment Tool (TRUST) [2], which was developed by our research group, classified uncertainty according to its source and model aspects. Sources are transparency, methods, imprecision, bias and indirectness, and unavailability. Model aspects are context/scope, model structure, selection of evidence, inputs, implementation, and outcomes. In addition, the concept of uncertainty is related to concepts such as risk, validity, quality, and relevance. In the context of reimbursement decision making, risk can be defined as the probability of making a wrong reimbursement decision combined with the consequences of making that decision. Model validity is defined as a model’s ability to reproduce reality [13]. In the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) system, quality of evidence is equated to the level of confidence in the results of the research [14]. The relevance of a model relates to how closely the problem addressed in the model applies to the problem faced by decision makers [15].

There are methods and method guides for performing assessment of uncertainty and related concepts [1, 16,17,18]. To make an uncertainty known that was previously unknown, it is necessary to identify all uncertainty in an HEE [2]. Only then can an uncertainty be analysed and communicated to the involved stakeholders. A comprehensive overview, including methods at the different stages of uncertainty assessment (identification, analysis, and communication), is lacking. In this article, we aimed to comprehensively review methods for uncertainty identification, analysis, and communication for use in model-based HEEs.

2 Methods

We performed a snowballing review in which we identified articles that may be relevant to the research question by screening references and citations as described by Wohlin [19]. We chose eight articles [1, 2, 9,10,11, 13, 15, 20] as starting points for the snowballing search. These articles were chosen because we considered them to be key papers for the topic of uncertainty assessment in HEEs. We conducted two snowballing iterations. For the first iteration we used both reference and citation searches of the initial eight articles, and for the second iteration we used only a reference search of articles that we identified in the first iteration. We conducted the snowballing searches in the Web of Science, Google Scholar and Embase databases on 18 January 2022 using their ‘cited by’ functions and lists of references. The relevant references were then imported for screening through Web of Science and deduplicated automatically there (further details about the literature search are reported in Appendix 1).

We included methodological articles that explicitly described methods to identify, assess or communicate uncertainty for use in model-based HEE. We excluded non-English and non-peer-reviewed articles, as well as articles that were unrelated to HEE. We extracted data on authors, year, and aim/description of the method as presented in the article.

All identified articles were subject to title and abstract screening and, if eligible, to a full-text review by one of the authors (TO). In case of ambiguity, we discussed the inclusion of articles with all authors. We categorised the identified methods into three categories, according to their stage of use: uncertainty identification, analysis, and communication. Furthermore, we subdivided the methods according to their purpose, which emerged from the review; these purpose groups were discussed with all co-authors.

Additionally, we classified all uncertainty identification methods into the sources of uncertainty that they covered following the TRUST tool. TRUST describes the following sources of uncertainty: transparency in the reporting, the appropriateness of methods used imprecision, bias and indirectness, and unavailability in the evidence used to develop the model. We considered that a method addressed a source of uncertainty when the method could be used to identify at least a partial lack of knowledge related to that source: for example, a checklist may ask a precise question about a common cause of intransparencies but not ask about all possible intransparencies, and we still considered that this checklist would cover transparency. This means that when a method addresses a source of uncertainty (Table 1), the method can help to identify uncertainties from that source, but not necessarily all uncertainties related to that source. The classification was conducted through extensive discussions between the authors.

3 Results

3.1 Literature Search

The search strategy identified 6018 references in total. After deduplication, a total of 3551 references were subject to title and abstract screening (Fig. 1). The first iteration identified a total of 5040 (2945 after deduplication) articles, and 150 were selected for full-text review. Of these, we included 34 articles (+8 starting point articles). Through a reference search for the second iteration of the snowballing review, we found 954 references (606 after deduplication). After title and abstract screening, 99 articles were selected for full-text review, of which 13 met the inclusion criteria. Together with the eight starting point articles, this adds to a total of 55 included articles. Within these articles we identified 80 methods, of which we categorised 30 as uncertainty identification methods, 28 as uncertainty analysis methods and 22 as uncertainty communication methods (further details on identification and communication methods are reported in Appendices 2 and 3). Figure 2 gives an overview of the stages of uncertainty assessment and the broad aims of methods that we have identified. A more complete figure including all methods can be found in Appendix 4.

Fig. 1
figure 1

PRISMA diagram. PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Full size image
Fig. 2
figure 2

Overview of identified uncertainty assessment methods

Full size image

3.2 Uncertainty Identification Methods

The majority (19) of the 30 identification methods assessed model quality and transparency of reporting [2, 15, 20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35, 46]. Three methods specifically aimed at determining the model validity [36,37,38] and eight at assessing the quality of evidence used to develop the model [14, 39,40,41,42,43,44,45] (Table 1). Methods to assess model quality, transparency and validation were all specific to health economic modelling, whereas methods to assess quality of evidence were either targeted to examine evidence in general [14, 45] or to examine the quality of specific types of research design such as indirect treatment comparisons and network meta-analyses [39], systematic reviews [43], observational studies [40, 42], non-randomised studies [44], or real-world evidence [41]. Knowing the quality of the evidence used to develop an economic model is essential for assessing the uncertainty in the model outputs. With the exception of TRUST [2], none of the methods were created with the aim of identifying uncertainties. As we used TRUST [2] (classified as model quality and transparency of reporting method) as a framework for categorising methods, TRUST considered all sources of uncertainty. With regard to sources of uncertainty, most methods could identify uncertainties caused by lack of transparency (26) [2, 15, 20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36, 39,40,41,42,43,44, 46] and/or methodological uncertainty (18) [2, 15, 20,21,22,23,24,25,26,27,28, 30, 33,34,35,36,37,38]. Besides TRUST, imprecision was considered by five quality-of-evidence methods [39,40,41,42,43,44] and unavailability was considered by two quality-of-evidence methods [39, 42].

Table 1 Uncertainty identification methods
Full size table

Descriptions and details about how each individual method was categorised can be found in Appendix 2.

3.3 Uncertainty Analysis Methods

We identified 28 uncertainty analysis methods, 16 of which were variations of one of the following five overarching methods: expected acceptability analysis [1], deterministic sensitivity analysis (DSA) [1], importance analysis [53], VOI analysis [9], and opportunity loss analysis [56, 57]. We subdivided the uncertainty analysis methods into four groups according to their purpose: quantify uncertain inputs when quantitative estimates are unavailable or indirect (2); generate base-case results using the most plausible assumption (7); explore uncertainty not included in the base-case (11); analyse consequences of uncertainty such as risk and VOI (8). All methods in the latter three groups (generating base-case results, uncertainty exploration, consequences of uncertainty) can analyse uncertainties from all sources if uncertainty was quantified. Only threshold analysis can be used to analyse an uncertainty when this uncertainty is not quantified. Thirteen methods require a probabilistic analysis (expected acceptability analysis [1], analysis for policy acceptability [49], fuzzy expected acceptability [50], probabilistic DSA [52], all six VOI methods [9, 55], HTA risk analysis [57], expected loss analysis [56], and real-options analysis [59]).

3.4 Uncertainty Communication Methods

We identified 22 uncertainty communication methods, nine of which were a variation of one of the following four overarching methods: cost-effectiveness planes [60], efficiency frontiers [63], expected acceptability curves [65], and tornado diagrams [1] (see the expanded table in Appendix 3). We used the same subdivisions as for analysis methods, except that there were no communication methods for the ‘quantification of uncertainty’ group. The groups of uncertainty communication methods according to their purpose were communicate base-case results (11), communicate the results of uncertainty exploration (5), and communicate the consequences of uncertainty (6). All uncertainty communication methods, except for the Assessment of Risk Table (ART) and Assessment of Risk Chart (ARCH), can only include uncertainty from all sources when the uncertainty is quantified. The majority (17) of the communication methods require probabilistic analysis. Only the methods range of results [1], standard tornado diagram [1], stepwise tornado diagram [52], distributional tornado diagram [52], and threshold analysis figure [1] do not require a probabilistic analysis.

4 Discussion

We identified 80 uncertainty assessment methods, of which 30 were uncertainty identification, 28 were uncertainty analysis, and 22 were uncertainty communication methods. All but one uncertainty identification method focus on specific sources of uncertainty. Nearly all methods for uncertainty analysis and communication require uncertainties to be quantified and the majority (30 of 50) also require inclusion of uncertainty in a probabilistic analysis. Of note, most of the uncertainty analysis and communication methods serve different purposes and are therefore not interchangeable. Our categorisation, division by purpose and description can help provide clarity as to what method may be appropriate for the desired purpose and stage of assessment. The desired purpose may depend on local requirements, for example guidance by health technology assessment (HTA) agencies. The analysis method used also depends on whether the uncertainty was quantified or not.

Briggs et al. [1] and Bilcke et al. [18] also provide an overview of uncertainty assessment methods. Briggs et al. [1] presented seven uncertainty analysis and five communication methods, describing their use and best practices in detail. Bilcke et al. [18] provided a flowchart with steps of uncertainty analysis depending on the type of uncertainty faced. Our article complements this literature by presenting an up-to-date and comprehensive review of 80 uncertainty assessment methods. In addition, our categorisation of methods by stage of assessment (identification, analysis and communication of uncertainty) and purpose may help in further clarifying the use of these methods within uncertainty assessment. Furthermore, we highlight which sources of uncertainty are covered by the different uncertainty identification methods (Table 1). This classification can be used to guide the selection of identification methods. The overlap in sources of uncertainty treated throughout different categories of uncertainty identification methods highlights the lack of conceptual clarity of different concepts.

Our study has some limitations. The starting point papers were predominantly focused on the field of HTA and health economics. We note that we attempted a broader search and review and abandoned it because it was not feasible, due to both the much larger search and review required and because this would also require an assessment of how these potentially additional methods could possibly be applied in the present context. We consider that this could be another potentially interesting article.

For feasibility reasons, we chose a snowballing review approach, and although 80 methods were identified, relevant methods may have been missed. Another challenge was that methods classified as uncertainty identification methods in the current review included methods that were developed to assess concepts such as validity, model quality, quality of evidence or model transparency. These were also able to identify uncertainties. Similarly, methods to assess uncertainties can identify issues regarding validity or model quality. This highlights that these concepts are related and that their boundaries are not always clear. Therefore, we had to review a broad range of literature to identify potentially useful methods for uncertainty assessment in health economic models. The lack of conceptual clarity for terms relating to uncertainty complicates methodological research in this area [14].

Another limitation of this study is the subjectivity in the classification of identification methods according to the sources of uncertainty that they assess. To mitigate this, all classifications were extensively discussed between the authors to obtain consensus on potentially ambiguous classifications. Because there was no objective threshold at which a source of uncertainty could be considered ‘fully’ covered, we considered a source of uncertainty to be covered even if this was only partially the case. As a result, in practical use the uncertainty identification methods may not fully cover some sources of uncertainty.

The comprehensive set of methods presented in Tables 1, 2 and 3 can help analysts to select methods for an uncertainty assessment, and decision makers in the interpretation of results, by giving them a brief explanation of how a method works.

Table 2 Uncertainty analysis methods
Full size table
Table 3 Uncertainty communication methods
Full size table

Within the group of identification methods, the quality and transparency of reporting methods are mostly checklists with similar aims, but different levels of detail and comprehensiveness of sources of uncertainty that they may identify. The proposed framework for quality assessment of decision-analytic models by Philips et al. [20], and CHEERS 2022 [27], have a high level of detail, with the former being focused on model-based evaluations and the latter not focused on model-based evaluations. CHEERS has been widely endorsed as a reporting tool for HEEs. The detail of the items of CHEERS makes it easy to use for less experienced analysts and improves reproducibility. However, the framework by Philips et al., and CHEERS 2022, only cover two sources of uncertainty: transparency and methods. TRUST [2] is the most comprehensive tool that we found, covering all sources of uncertainty, but it does not contain detailed questions about each item. This shows that checklists for uncertainty identification are complementary, and we recommend that both level of detail and comprehensiveness are considered when selecting methods.

Within the categories of methods for uncertainty analysis and communication, the selection of a method will depend on the specific aim and audience of the analysis. For instance, the National Institute for Health and Care Excellence (NICE), National Health Care Institute (Zorginstituut Nederland; ZIN) and Health Information and Quality Authority (HIQA) require the use of probabilistic analyses, cost-effectiveness acceptability curves, simple tornado diagrams, and scenario analyses [17, 68, 69]. Furthermore, selection of methods will depend on whether uncertainty is expressed as a probability, and constraints in time and skills of the analysts. We consider that an application and comparison of methods, with the outlining of advantages and disadvantages, could be an interesting topic for further research. However, we note that most analysis and communication methods serve different purposes and can therefore not be used interchangeably.

Additionally, further research is necessary to analyse and communicate uncertainty that is not quantified. We found only two methods to analyse uncertainty that is not quantified (expert elicitation, threshold analyses [1]) and three methods to communicate uncertainty that is not quantified (the threshold analysis figure [1], the ART [67] and the ARCH [67]). Threshold analyses and the relating figure result in an estimate of how sensitive model outcomes are, when uncertainty is not quantified. Both ART and ARCH ask for analysts to state information about unquantified uncertainty, risk, and their impact on decision making. Due to imperfect evidence and challenges in the quantification of uncertainties, it is unlikely that all uncertainties will ever be quantified [5].

Structured expert elicitation can help to quantify previously unquantified uncertainty. This allows these types of uncertainties to be included in analysis and communication methods when this may otherwise not be possible [70]. Moreover, compared with other methods of obtaining new evidence, expert elicitation can be used to quickly and efficiently quantify inputs and uncertainty [71]. While expert elicitation is a versatile method, it is limited by the availability of experts with sufficient expertise [71]. Furthermore, HTA agencies making reimbursement decisions based on model-based economic analyses offer limited guidance on the conduct and reporting of expert elicitation [69, 72]. As a possible result of this, a review of company submissions to NICE found significant variations in the quality of conduct and reporting of expert elicitation [73]. The endorsement and use of reporting guidelines [70, 74, 75], developed specifically for structured expert elicitation in HTA, may help improve quality and transparency.

All methods except for expert elicitation rely on implicit judgements: the interpretation of the results of threshold analyses/figures requires decision makers to make judgements on the likelihood of the threshold being reached, and ART and ARCH require analysts or decision makers to state information on unquantified uncertainty. Current assessment of uncertainties that are not quantified are therefore not transparent. Furthermore, detailed guidance on how to assess unquantified uncertainty by reimbursement agencies are currently missing. Further research is necessary to find out how to address unquantified uncertainty adequately and transparently.

5 Conclusion

Uncertainty assessment methods exist that can address uncertainty from different sources. The overview presented here will help analysts and decision makers in the choice of methods for uncertainty assessments. Quantification of uncertainty is key to uncertainty analysis and communication. Where not all uncertainty is quantified, efforts need to be undertaken to analyse and communicate the unquantified uncertainty and resulting consequences. Further clarification of terminology and guidance on the use of (combinations of) methods to identify uncertainty, and related concepts such as validity and quality, could improve uncertainty assessment.