1 Introduction

The pace of innovation in healthcare is accelerating, and society demands innovative healthcare programmes to become available as soon as possible. Therefore, decisions regarding the use and reimbursement of new programmes need to be made when scientific evidence is by definition still scarce and effectiveness and cost effectiveness are (highly) uncertain. Dealing with uncertainty in healthcare decision making is challenging. First, since not all uncertainties are easily incorporated, generally only some uncertainty is explicitly characterised and presented in assessments [1]. As a consequence, researchers focus on quantifiable uncertainties such as measurement error. The logic of quantification, or “mathematics of certainty” [2], may result in ignoring important uncertainties which are hard or impossible to quantify (such as degree of generalisability of effectiveness to different populations and long-term consequences), causing pseudo-certainty of the results generated. Second, decision makers traditionally resort to science for certainty. Being transparent about uncertainty conflicts with this notion of scientific certainty and is perceived by decision makers as complicating decision making.

At the same time, the steep rise in healthcare expenses, combined with the ongoing economic crises, necessitates painful and far-reaching decisions on the allocation of healthcare resources. These decisions are often based on model-based assessments of the comparative (cost)effectiveness of the innovative programme [3]. To account for timely and optimal decisions in the long run, it is crucial that all relevant uncertainties are incorporated in the assessments [4]. These uncertainties need to be identified as soon as possible, before the assessment is performed. This allows for transparent decision-making processes leading to defensible decisions and maintaining or increasing public trust in research results as well as policy-making bodies [5]. Moreover, allowing relevant uncertainties to be neglected in the assessments may stimulate companies to not collect evidence, in order to increase the chances of reimbursement for their product [6]. What is—and, hence, also what is not—assessed sets the agenda and frames the debate [4].

In this commentary, we argue that knowledge and concepts on dealing with uncertainty from other scientific disciplines, such as environmental science and risk governance, can improve the handling of uncertainty in healthcare. With this commentary, we hope to raise awareness of the current challenges with acknowledging uncertainty in healthcare policy decision making, and to motivate researchers and decision makers to become more tolerant towards uncertainty.

2 Defining Uncertainty

In the field of healthcare, various types of uncertainty are distinguished in the assessment of innovative programmes: parameter uncertainty, structural uncertainty, methodological uncertainty, variability, heterogeneity and decision uncertainty [7]. Decision uncertainty is the umbrella term for all uncertainty surrounding a decision, and can be caused by any other type of uncertainty. Parameter uncertainty has received most attention, both in theory and in practice [1, 8]. It relates to the fact that the true value of a parameter is not known [8]. In practice, it mostly refers to imprecise estimates and standard errors surrounding a mean value, which corresponds to measurement error. Methodological uncertainty can be defined as disparities in the choice of analytic methods that underpin an assessment (e.g. the choice of perspective) [8]. Structural uncertainty refers to uncertainty surrounding the structure of a decision model. Variability relates to the fact that individuals are unique and therefore vary in their outcomes, which may partly be explained by individual characteristics (heterogeneity) [9].

Outside the field of healthcare, Walker et al. [10] define uncertainty as “any deviation from the unachievable ideal of completely deterministic knowledge of the relevant system”. They distinguish four levels of uncertainty (Fig. 1). Statistical uncertainty is the imprecision, or sampling error, that can be described adequately in statistical terms [10]. For scenario uncertainty there is a range of possible outcomes, but the probability of a particular outcome occurring cannot be formulated. Recognised ignorance represents more fundamental uncertainty, where we do not even know the range of possible outcomes. Here, the scientific basis for developing scenarios is weak. While in recognised ignorance we know what we do not know, total ignorance implies a deeper level of uncertainty, to the extent that it is unknown what is unknown. In the present paper scenario uncertainty and (recognised or total) ignorance are referred to as non-statistical uncertainty.

Fig. 1
figure 1

Different levels of uncertainty, ranging from statistical uncertainty to total ignorance (adapted from Walker et al. [10]). The shading represents the acknowledgement of these levels of uncertainty in the assessment of healthcare programmes: the darker the colour, the more this level of uncertainty is acknowledged. The continuing arrow is used to indicate that there is no way of knowing the full extent of ignorance

Full size image

In healthcare, the Centers for Medicare and Medicaid Services (CMS) recently decided not to reimburse positron emission tomography (PET) imaging with amyloid-β (Aβ) in patients with suspected dementia or neurodegenerative disease [11]. The assessment focused on statistical uncertainty surrounding the accuracy of the test, mostly due to small sample sizes. However, there was no evidence of the resulting health benefits or harms, or the sustainability of results outside the clinical trial setting, in broad community use. This example is typical in that assessments in healthcare focus on statistical uncertainty (the sampling error associated with the accuracy of the test), rather than scenario uncertainty (e.g. differences between a controlled clinical trial and the real world) or recognised ignorance (e.g. harms of the test for the patient).

3 Dealing with Uncertainty

In the theoretical decision framework for reimbursement of innovative healthcare programmes, three key questions need to be answered (Fig. 2) [3, 12].

Fig. 2
figure 2

Schematic representation of the decision-making framework (adapted from Chalkidou et al. [3]). Uncertainty is considered in questions 2 and 3

Full size image

The first answer is based on the expected benefit of a programme, given existing information. Regardless of the choice of benefit, the underlying assumption is that all relevant inputs are quantifiable, hence resulting in a similarly quantifiable expected benefit. However, this is usually not the case, as innovation is actually an important generator of non-statistical uncertainty, which is only partly quantifiable [13]. Existing theoretical frameworks may therefore provoke researchers to omit non-statistical uncertainty, causing pseudo-certainty. In the second and third question of the decision-making framework, the costs of decision uncertainty are calculated, and the benefits of postponing reimbursement are weighed against its costs in terms of health benefits forgone [14, 15]. This again requires that the probability of an outcome occurring is formulated, which is only possible for statistical uncertainty.

An example of a framework from another field is the risk governance framework of the International Risk Governance Council. This framework clearly distinguishes between analysing and understanding uncertainty (appraisal), and deciding what to do with it (management) [16]. Before an assessment is performed in the appraisal phase, different stakeholders discuss the potential consequences (positive or negative). This may reveal potential scenario uncertainty or recognised ignorance that would otherwise have been neglected, and informs the subsequent assessment. The management phase explicitly includes monitoring the consequences of the decision and reviewing the decision if necessary. Translating this to healthcare, scoping activities can be used to discuss with all stakeholders which uncertainties are considered relevant, what the associated consequences may be, and how these can be incorporated in an assessment. Such ‘mapping the decision situation’ informs the assessment, and might encourage stakeholders to acknowledge the necessity of information about uncertainty [17]. Although initiatives of early dialogue already exist in healthcare, they do not explicitly address the issue of uncertainty [18]. In the recent Aβ PET example, the CMS decided to cover the imaging technique only in clinical studies [coverage with evidence development (CED)] [11]. Because these studies must be approved by the CMS, they can see to the collection of evidence that is relevant for their decision. However, this implies knowledge about which uncertainties are relevant for the decision. If the impact of all existing uncertainties had been formally explored in the assessment (appraisal), e.g. using scenario analysis, this information could have been used in developing the CED (management), informing the consequences of reimbursement and reviewing the decision if more evidence becomes available.

4 Discussion

To account for timely and optimal decisions in the long run, it is crucial that all relevant uncertainties are incorporated in the assessments. This does not mean that we argue against evidence-based decision making. We argue that evidence is more than just the empirical data, preferably from randomized studies, decisions are now generally based on. The limits to knowledge need to be uncovered, rather than assuming existing knowledge to be correct [19]. Moving towards uncertainty acceptance, and hence the production of uncertainty information in the assessment of innovative programmes, is an important step in this direction.

However, two critical challenges can be identified. First, the current role of non-statistical uncertainty in healthcare decision making is unclear. If there is a wish to develop towards a scientific and political community in which careful and transparent handling of all relevant uncertainties is standard practice, new frameworks for healthcare decision making upon innovations, as well as tools to handle non-statistical uncertainty in the assessments, are needed. Careful evaluation of the feasibility and usefulness in healthcare of future-oriented assessment approaches, such as formal scenario analysis or other foresight methods, may be a way forward [20, 21].

5 Conclusion

As Sir Iain Chalmers wrote ten years ago, “A prerequisite for constructive debate about uncertainties about the effects of treatments is a greater willingness among professionals and the public to admit and discuss them” [22]. Food for thought can be found in other fields of research, and, vice versa, innovative methodology developed in the field of healthcare might contribute to other decision-oriented research fields. In the end, accepting uncertainty will not make the inevitable decisions in healthcare less painful, but will help to better allocate the scarce healthcare resources, and make these decisions more accountable and, therefore, acceptable.