The IPCC AR5 guidance note on consistent treatment of uncertainties: a common approach across the working groups

The AR5 Guidance Note instructs an author team, when writing a chapter of an IPCC assessment, to describe and evaluate the state of knowledge related to topics under its purview: particularly, the type, amount, quality, and consistency of evidence and the degree of agreement.

Types of evidence include, for example, mechanistic or process understanding, underlying theory, model results, observational and experimental data, and formally elicited expert judgment. The amount of evidence available can range from small to large, and that evidence can vary in quality. Evidence can also vary in its consistency, i.e., the extent to which it supports single or competing explanations of the same phenomena, or the extent to which projected future outcomes are similar or divergent.

The degree of agreement is a measure of the consensus across the scientific community on a given topic and not just across an author team. It indicates, for example, the degree to which a finding follows from established, competing, or speculative scientific explanations. Agreement is not equivalent to consistency. Whether or not consistent evidence corresponds to a high degree of agreement is determined by other aspects of evidence such as its amount and quality; evidence can be consistent yet low in quality.

The assessment of the state of knowledge for topics covered in AR5 chapters will necessarily provide descriptions of relevant evidence and agreement. For each topic, the description should summarize the author team’s evaluation of evidence and agreement for that topic and any conclusions the author team reaches based on its assessment. It should also present the range of relevant scientific explanations where a diversity of views exist. For example, the author team could summarize the literature published on a topic and its evaluation of the scientific validity of potential explanations for the information available.

A key finding is a conclusion of the assessment process that the author team may choose to include in the chapter’s Executive Summary. The author team’s evaluation of evidence and agreement provides the basis for any key findings it develops and also the foundation for determining the author team’s degree of certainty in those findings. The description of the author team’s evaluation of evidence and agreement is called a traceable account in the Guidance Note. Each key finding presented in a chapter’s Executive Summary will include reference to the chapter section containing the traceable account for the finding.

The AR5 Guidance Note lists commonly encountered categories of findings, providing author teams with instructions for assigning calibrated uncertainty language for each category. These criteria, described in paragraph 11 of the Guidance Note, revise the list of quantitative categories presented in the AR4 Guidance Notes (IPCC 2005). The AR5 Guidance Note attempts to make the instructions more broadly applicable across all three Working Groups, describing how summary terms for evidence and agreement, confidence, likelihood, and other probabilistic measures should be used to characterize findings in different situations. The AR5 Guidance Note also explicitly removes the concept of this listing being a “hierarchy.” It instead indicates that calibrated uncertainty language should be used to provide the most information to the reader and should reflect the nature of the available evidence and agreement, which varies by discipline and by topic.

The AR5 Guidance Note, as in the TAR guidance paper (Moss and Schneider 2000) and AR4 Guidance Notes (IPCC 2005), contains further, more general advice for effectively constructing key findings of the assessment process and for communicating their degree of certainty. Author teams are encouraged to be aware of group dynamics that can affect the assessment process, to consider ways to phrase findings clearly and precisely, and to consider all plausible sources of uncertainty. Additionally, the AR5 Guidance Note more extensively encourages author teams to provide information on the range of possible outcomes, including the tails of distributions of variables relevant to decision-making and risk management. Finally, the AR5 Guidance Note describes formulation of conditional findings, for which the degrees of certainty in findings (effects) conditional on other findings (causes) can be evaluated and communicated separately. Construction of conditional findings can combine information from multiple chapters within one Working Group or across Working Groups.

The consideration of uncertainties is key to the comprehensive and robust assessment of the “Physical Science Basis of Climate Change” by Working Group I of the IPCC. Working Group I has successfully applied the calibrated IPCC uncertainty language in past assessment reports. In fact, the quantification of uncertainty ranges in experimental and theoretical science in general, and in climate science in particular, is one of the central tasks of the scientific activity. The common approach is to report this in an uncertainty estimate based, e.g., on the statistical measure of the standard deviation. However, the challenge when writing IPCC assessment reports is to translate this numerical information into text that is widely understandable and is interpreted correctly.

Since the TAR, when a first version of an IPCC guidance document on the treatment of uncertainties (Moss and Schneider 2000) was made available to the authors, key scientific statements in Working Group I’s contribution have been accompanied by the assessment of associated uncertainty and communicated using calibrated uncertainty language, with levels of confidence to express expert judgments on the correctness of the underlying science and likelihood statements to indicate the assessed likelihood of an outcome or a result. This approach was then further expanded on in the AR4, using the IPCC uncertainties guidance paper for the AR4 (IPCC 2005; Manning 2006), and key statements from the TAR were updated accordingly. The currently underway Working Group I contribution to the AR5 will use the by now well established approach as laid out in the revised AR5 guidance paper (Mastrandrea et al. 2010).

3.1.2 Example of a complex, multi-step process to arrive at a synthesis statement of uncertainty

Even though the physically based climate sciences offer various mathematical methods to determine and estimate uncertainties, overall statements require ultimately some amount of expert judgment. A good example for this complex assessment process is the range of projected global mean temperature increase for a selected emissions scenario (Meehl et al. 2007). The quantitative information comes from multiple lines of independent evidence, based on simulations using a hierarchy of climate models and different experimental set-ups as illustrated in Fig. 3. Using a vast range of climate model simulations, individual physics tests, and idealized simulations from models including additional processes such as the global carbon cycle, each approach provides an outcome or range of the increase of global mean temperature by the end of the 21st century. All these estimates are obtained from formal calculations applying well-defined and well-established mathematical and statistical techniques. A synthesis of the results for each scenario is shown in Fig. 4.

Open image in new window

Open image in new window

The challenge of the assessment process is to synthesize this information into one uncertainty range. Due to the variety, different quality and origin of the information, a formal procedure is not available and expert judgment is called for in the synthesis, the last step of the assessment of uncertainty. The result of this final expert judgment is the −40% to +60% range shown as the grey bars in Fig. 4. This range provides an uncertainty range which captures the basic spread of the model projections when all the combined information is considered. More importantly, the asymmetry of the uncertainty range conveys the important message that uncertainty is larger at the upper end of the temperature range. This result is based on process understanding stemming from carbon cycle simulations and expresses the consensus that the global mean feedback, when including the carbon cycle response in climate models, is positive.

Meehl et al. (2007) therefore concluded in the Executive Summary of Chapter 10 of Working Group AR4 that “[A]n assessment based on AOGCM projections, probabilistic methods, EMICs [Earth System Models of Intermediate Complexity], a simple model tuned to the AOGCM responses, as well as coupled climate carbon cycle models, suggests that for non-mitigation scenarios, the future increase in global mean SAT [surface air temperature] is likely to fall within −40 to +60% of the multi-model AOGCM mean warming simulated for a given scenario. The greater uncertainty at higher values results in part from uncertainties in the carbon cycle feedbacks”. The best estimates and asymmetric likely ranges of projected global mean temperature increase for the SRES non-mitigation scenarios were elevated to both Technical Summary and SPM of Working Group I AR4, and further highlighted in key figures and tables of both documents. A more extensive discussion about these issues can be found in the review by Knutti et al. (2008). It is important to recognize that the ultimate expert judgment (Fig. 3) is only made possible by the large amount of model-based information which resulted from a concerted and timely effort of the climate modeling community.

The Working Group II contribution to the AR5 spans a wide spectrum of topics, with expanded coverage of climate change impacts and adaptation as compared to the AR4 contribution (IPCC 2007b). New chapters will focus on oceans, on additional dimensions of human systems, and on further aspects of adaptation. Impacts will be considered more extensively under multiple scenarios of climate change and of socioeconomic development, as well as in the context of adaptation and mitigation strategies, particularly at the regional scale. The contribution will include expanded consideration of the interaction among climate change and other stressors. Throughout, the contribution will emphasize information to inform risk management decisions, particularly information on the range of possible outcomes, including the tails of distributions where probabilistic information is available.

The Working Group II contribution will assess knowledge derived from many disciplines, and the nature of relevant uncertainties will vary, requiring different approaches for evaluating the degree of certainty in key findings. All chapters will be encouraged to describe evaluations of evidence and agreement with traceable accounts, including discussion of the range of relevant scientific interpretations of the evidence available. Traceable accounts will constitute a fundamental component of each chapter’s assessment, communicating the author team’s conclusions and the evaluation of evidence and agreement underpinning those conclusions. Based on their evaluations of evidence and agreement, author teams will be able to choose to express their degree of certainty in findings using summary terms for evidence and agreement, levels of confidence, or likelihood terms or other probabilistic measures, using the AR5 Guidance Note.

Constructions of conditional findings are expected to be particularly relevant for the Working Group II contribution. Conditional findings as introduced in the AR5 Guidance Note can separately characterize the degree of certainty in causes and effects. For example, a Working Group II author team could construct a finding about impacts or adaptation that is conditional on a Working Group I finding about changes in the physical climate system. A finding about potential changes in the physical climate system can describe expected directions or magnitudes of change, providing information about possible climate outcomes such as best estimates, ranges, or relative likelihoods. Presented climate outcomes will have implications for impacts and adaptation, and thus findings about the physical climate system, impacts, and adaptation are linked. The degree of certainty in a finding about impacts or adaptation, however, can be evaluated separately from the degree of certainty in a finding about the climate outcome to which it is linked. A finding about impacts or adaptation will be based on evaluation of evidence and agreement regarding the consequences of a given change in climate, which is independent from evaluation of whether the change will occur or not. In this way, a finding regarding impacts of or appropriate adaptation responses to a climate outcome (were it to occur) could have a high degree of certainty even if the degree of certainty in the climate outcome itself is low. Conversely, a finding regarding impacts or adaptation could have a low degree of certainty even if the degree of certainty in the climate outcome is high.

Throughout the Working Group II contribution, findings will require specification of underlying assumptions on which impacts and adaptation outcomes are dependent. For example, projected impacts are based not only on projected changes in climate, but also on assumptions about trajectories for socioeconomic development and their implications for vulnerability to climate changes. Figure 5 illustrates the linkages among impacts and adaptation outcomes conditional on a hypothetical range of climate outcomes and on one set of socioeconomic assumptions. Each combination of climate outcome and socioeconomic assumptions will have different implications for impacts and adaptation, but there may also be commonalities across such combinations that emerge, for example when a given impact or adaptation outcome is projected for a range of possible climate outcomes. Impacts or adaptation findings need to communicate clearly the climate and socioeconomic assumptions on which they are based and where such commonalities exist. It is important to distinguish, for example, between impacts or adaptation outcomes associated with only one future climate and socioeconomic scenario and outcomes that arise under a range of future scenarios.

Open image in new window

The Guidance Note mentions the importance of developing findings that synthesize across underlying lines of evidence, providing general, substantive conclusions that appropriately reflect the available evidence. Along these lines in the Working Group II contribution, an author team may consider constructing a key finding that extends beyond the spatial or temporal scale of available evidence. That is, an author team may consider a finding about a region based on evidence for several countries within that region. Such generalizations beyond the scale of available evidence affect the degree of certainty in those findings, and author teams will need to consider and communicate assumptions made.

The Working Group II contribution to the AR5 will employ all forms of calibrated uncertainty language presented in the AR5 Guidance Note: summary terms for evidence and agreement, as well as the more clearly distinguished confidence and likelihood metrics. With the confidence metric qualitatively calibrated, it is expected to be broadly applicable across the Working Group II contribution for findings where probabilistic information is not available, for example, for findings related to adaptation and for findings that qualitatively project future outcomes. Where probabilistic information is available, likelihood terms or more precise probabilistic measures will be used, with an emphasis on providing as much information as possible on the full distribution of possible outcomes, including the tails, to inform policy decisions for risk management. In all cases, it is important that findings in the Working Group II contribution explicitly communicate their degree of certainty, specify assumptions appropriately, and be clearly supported by traceable accounts in the underlying chapter text.

The Working Group III contribution to the AR5 will explore a wide range of available options and instruments for greenhouse gas mitigation. The outline of the Working Group III contribution to the AR5 foresees a detailed assessment of options for reducing land use (change) emissions and for transforming energy systems, energy use of households, industry, transport and other sectors. It will also include a review of the emerging literature on geoengineering options such as managing the carbon cycle and altering global radiation balance.

In order to provide a comprehensive and consistent assessment of mitigation options available to society and decision makers, the IPCC has initiated a dialogue with several scientific communities in the areas of climate science, mitigation, and adaptation. These communities are engaging in a coordinated scenario-building process where the IPCC no longer commissions specific scenarios, but restricts its role to facilitating the scenario process which is conducted by self-organized bodies of the scientific communities (Moss et al. 2010). The new scenarios are characterized by combining (i) levels of radiative forcing (Representative Concentration Pathways - RCPs) (van Vuuren et al. 2011) with (ii) a set of socioeconomic reference assumptions that specify future developments for key parameters such as GDP and population (van Vuuren et al. 2010; Kriegler et al. 2010).

In order to be policy relevant without being policy prescriptive, an ex-post analysis of these scenarios has to take into account and make explicit (i) underlying value judgments of scenarios and (ii) the interaction between ends and means, i.e., how the direct and indirect consequences of the development of mitigation options affects the assessment of the ends (e.g., global stabilization targets) and vice versa. For example, low-stabilization scenarios might imply an extensive use of biomass which may be associated with serious conflicts over land-use and trade-offs between clean energy and low-cost food supply. The IPCC intends to assess multiple scenarios describing different plausible future pathways for mitigation. These multiple scenarios should also include second best scenarios which explicitly consider, for example, the limited availability of technologies, fragmented carbon markets, or suboptimal policy instruments.

Beyond these ex-post analyses pointing out the options decision-makers might have at their disposal, a well-designed uncertainty analysis is required to address epistemic uncertainties about the socioeconomic system. These epistemic uncertainties can be characterized by model uncertainty and parametric uncertainty.

Model uncertainty refers to different philosophies of representing socioeconomic and technology systems as well as value systems. For example, economic Partial Equilibrium models (such as energy system models) differ substantially from economic General Equilibrium models (such as Computable General Equilibrium Models). In general, model uncertainty may be reduced by eliminating some model structures from the set of plausible models. One way of doing so is validating models against empirical evidence to discriminate “better” models and consequently discard “bad” models. This requires, however, a clear idea about the relevant empirical evidence and the appropriate type of validation exercises—a particular difficult issue for policy evaluation models that commonly use counterfactual no policy cases in the evaluation process. But in the economic community, for example, there is no consensus regarding which stylized facts of past economic development should be reproduced by models. There is substantial disagreement about the relevant scale (global, regional) and the relevant time horizon (short-term, long-term) of stylized facts.

However, even “perfect validation” is only a necessary but not a sufficient condition for the selection of models. “Ockham’s razor” proposes that if a model explains the same empirical phenomena using less specific or more plausible assumptions and parameters than another model, then it can be deemed preferable (Edenhofer et al. 2006). Yet to this date, the theoretical and empirical foundation of model types within economics remains insufficient to allow for a consensus within the scientific community according to these principles. In other words, under the present state of the art, the uncertainties about the appropriate economic model structure would remain even if there is a consensus on the stylized facts.

A promising approach for explaining differences in model results is to conduct modeling comparison exercises of climate and energy economics models that trace the underlying drivers of different model results. Such exercises also enable evaluation and explanation of results when sensitivity analysis leads to the same qualitative results despite structural differences of models (e.g. most models used in climate economics find that more ambitious mitigation targets increase costs). The Joint IPCC Workshop of Working Group III and II on Socioeconomic Scenarios for Climate Change Impact and Response Assessments, held in November 2010, included discussion of the importance of modeling comparison exercises by the scientific community to enhance the understanding of underlying model philosophies and the robustness of the results.

Parameter uncertainty refers to a lack of empirical knowledge to calibrate the parameters of a model to their “true” values. Parameter uncertainty implies uncertainty in the predictions of any one model, and discrepancies may result even for otherwise very similar models when different assumptions about parameter values are adopted. Parameter uncertainty can be addressed in model specific uncertainty analyses including sensitivity analysis and Monte Carlo simulations. Examples for parameter uncertainty include technology costs, resource availabilities, or political decisions such as on international cooperation. Sensitivity analyses should not only be conducted on uncertain economic assumptions such as base rates of technological progress and price elasticities, but also across so-called second best worlds, in which certain assumptions on, for example, first best policy and market fragmentation or technology availability are varied to study their impacts on mitigation strategies. The evaluation of second best worlds typically reveals that such imperfections can have a pronounced effect on costs of mitigation (Clarke et al. 2009; Edenhofer et al. 2009; Edenhofer et al. 2010; Luderer et al. 2011). Conducting such studies is thus vital to represent policy options in a more realistic manner, and Working Group III intends to emphasize the assessment of such scenarios in AR5.

The new AR5 Guidance Note provides calibrated language for systematic evaluation and representation of these uncertainties. At a first level, both model uncertainty and parametric uncertainty can be characterized by the criteria of evidence and agreement. The Guidance Note indicates that evidence is generally most robust when there are multiple, consistent and independent lines of high-quality evidence. For example, the reference to modeling comparison exercises and to sensitivity studies should enable authors to trace back their assessments to scientific literature that aggregates findings from multiple independent modeling efforts. Admittedly, model comparison exercises will not resolve the problem of an incomplete representation of reality, as some feedback loops ignored in current models could become important in the future. In addition, the conventional validation procedure will not work in such a context. Modeling comparison exercises, however, can still be used as a social learning process to facilitate an ongoing discussion about feedback loops that might be important in the future, and they are valuable to justify the level of agreement within the scientific community. In addition, modeling comparison exercises and sensitivity studies may allow authors to qualitatively aggregate evidence and agreement into confidence statements. Even if the evidence base for quantitative evaluation of likelihoods in the context of Working Group III is limited, the AR5 will benefit substantially from qualitative explication of the uncertainties underlying the assessment of mitigation options.