The latest iteration of IPCC uncertainty guidance—an author perspective
- 2.4k Downloads
The latest iteration of Intergovernmental Panel on Climate Change (IPCC) uncertainty guidance is simpler and easier to use than the previous version. However, its primary focus remains assessing “what is at risk” under climate change, thus is most suitable for dealing with the scientific uncertainties in Working Group I and part of Working Group II findings. I distinguish between tame and complex risks, arguing that the guidance is most suited to assessing tame risks. Climate change is a complex risk, and as such as can be divided into idealized, calculated and perceived risks. While science has claims to objectivity, risk has a specific value component: when measuring gain and loss, calculated risks compete with risky options to manage those risks. The IPCC is charged with calculating risk (IPCC 2007, p22) but the communication of key findings takes place in an environment of competing perceived risks. Recommendations for managing this complex environment include separating scientific and risk-based findings, treating uncertainties for each separately; strengthening the philosophical basis of uncertainty management; application of a methodical scientific research program; clearly communicating competing findings, especially in the social sciences; and application of multiple frame to policy-relevant findings as reflected in the literature.
KeywordsCalculated Risk Uncertainty Management Current Guidance Core Theory Third Assessment Report
Human-induced climate change is a wicked problem beset by multiple risks. These risks are diverse and not necessarily commensurate in how they are framed by experts, decision-makers or the ordinary person (Giddens 2009; Hulme 2009). The risks of a changing climate can be framed in different ways. For example, measures of impact risks include assessing key vulnerabilities to help define anthropogenic interference with the climate system (Schneider et al. 2007); the market costs of economic impacts at market cost or change in future welfare measured as the social cost of carbon (Tol 2009; Watkiss 2011). Uncertainties associated with mitigating climate risks are similarly diverse. Various methods manage tangible and intangible values differently, and apply different rates of time preference and cultural values. Governance structures for implementing climate policy are also influenced by a range of views. These diverse frames introduce other types of uncertainty that extend well beyond scientific uncertainty.
The IPCC’s charter is to deliver policy-relevant advice. This advice is not restricted to scientific findings about future climate but also includes the impact of decisions made as a result of those findings. Such advice will be interdisciplinary rather than disciplinary in nature; therefore narrowly-focused uncertainty guidance is unlikely to be sufficient for decision-making needs.
A narrow focus will apply consensus to scientific findings but might also seek to frame policy-relevant advice in consensus terms. It may “play safe” relying on conventional interpretations of disciplinary approaches to managing uncertainty. A broad focus will describe the consensus scientific view, but will also communicate all scientific evidence, no matter how uncertain, that contributes to the plausible range of potential future climate risks. For policy, uncertainty arises where the same evidence is interpreted in different ways. Because of the difficulty in assessing a “best” policy in any objective sense, a range of policy positions may be valid. While a central goal of uncertainty guidance has been to overcome misinterpretation of the science, a consensus approach to framing policy-relevant advice is not appropriate for guidance contributing to diverse decision-making strategies.
This paper explores how the above issues relate to the recent uncertainty guidance prepared for the IPCC’s Fifth Assessment Report (AR5; Mastrandrea et al., 2010). It contrasts the issue of “wicked risks” in complex systems with tame risks faced in simpler situations. The current guidance does not distinguish between the communication of science and of risk, despite risk having a specific value component that measures benefit or harm. While this may be appropriate for tame risks, wicked risks are characterized by conflicts between scientifically calculated risks, different but valid ways to calculate those risks and perceived risks. For complex risks, there is a strong case for separating the risk analytic stage from the risk management stage, because of the uncertainty surrounding the decisions taken to manage those risks; i.e., both climate change and its management are risky.
The initial motivation for providing IPCC guidance on uncertainty was to develop common language to ensure the consistent communication of findings and improve the understanding of climate risk (Moss and Schneider, 2000). The use of unquantified terms in communication such as “likely”, “unlikely” and “probable” conveys overlapping and widely varying ranges of likelihoods. Furthermore, these ranges are perceived as being even wider by recipients of that communication (Morgan et al. 1990; Moss and Schneider 2000). Two further iterations of IPCC guidance have since been produced (IPCC 2005; Mastrandrea et al. 2010). Comprehensive guidance for uncertainty management has also been prepared by the US Climate Change Science Program (Climate Change Science Program 2009; National Assessment Synthesis Team 2001). The scientific component of this guidance is broadly consistent with the IPCC (they share common authors and origins) but the latter especially expands into cognitive challenges and decision-making. Other recent efforts have dealt with complex system issues, for example combining qualitative and quantitative uncertainty and incorporating reflexive learning (Van Der Sluijs et al. 2005; 2008).
The principal focus of the IPCC uncertainty guidance is on confidence and likelihood. The latest version provides qualitative guidance for confidence and quantitative guidance for likelihood. Uncertainty was not described consistently across the Working Groups during the Third Assessment Report (TAR), so that became a major aim of the AR4 guidance (Manning et al. 2004). The AR4 guidance provided quantitative scales for confidence expressed as odds and likelihood expressed as probabilities but qualitative descriptions for confidence were preferred by author teams. The two quantitative scales were criticized in the InterAcademy Review of the IPCC for being confusing and of limited use (Committee to Review the Intergovernmental Panel on Climate Change Science Program, 2010). The review recommended that the derivation of confidence and likelihood should be traceable, terms conveying likelihood be linked to quantitative ranges, confidence be qualitative, findings be well defined and that the precision of likelihood be influenced by level of confidence as outlined by Risbey and Kandlikar (2007).
Decision-making is framed as the response to scientific findings about risks under climate change, and the guidance material is most suited to communicating the analytic stages of risk assessment. This covers the results of Working Group I on climate science and Working Group II on impacts. However, the guidance is less suited to communicating the latter stages of risk management; namely, adaptation and mitigation. Some of the issues raised in the introduction: multiple frames, incommensurate values and how “good” policy can be distinguished from “poor” policy have received little attention. While the IPCC can assess the means to test policy benefits it cannot explicitly prescribe policy options—nor should it implicitly do so by framing adaptation and mitigation in specific ways. Lacking is guidance on how to assess and communicate competing frameworks for the risk assessment and management of climate change.
1.2 Scope of IPCC findings
The potential scope of findings can be explored via the IPCC’s terms of reference and through specific requests from the members of the IPCC for AR5. The role of the IPCC is to assess on a comprehensive, objective, open and transparent basis the scientific, technical and socio-economic information relevant to understanding the scientific basis of risk of human-induced climate change, its potential impacts and options for adaptation and mitigation. IPCC reports should be neutral with respect to policy, although they may need to deal objectively with scientific, technical and socio-economic factors relevant to the application of particular policies (IPCC 1998). With regard to policy, the following agreed paragraph from the recent United Nations Framework Convention on Climate Change (UNFCCC) Council of Parties meeting informs the assessment of adaptation policy: Affirms that enhanced action on adaptation should be undertaken in accordance with the Convention; follow a …, participatory and fully transparent approach, taking into consideration vulnerable groups, communities and ecosystems; and be based on and guided by the best available science and, as appropriate, traditional and indigenous knowledge; with a view to integrating adaptation into relevant social, economic and environmental policies and actions (Ad Hoc Working Group on long-term Co-operative Action 2010). Although this guidance originates from member countries (governments), the reference to actions which can be both public and private, are relevant to a wide audience. Here, the audience of IPCC reports is considered to be decision-makers at any level.
Since its formation the IPCC, UNFCCC and their member nations have been carrying out an iterative risk assessment and management exercise (IPCC 2007). The emphasis has progressively moved from scoping and analysis through to risk evaluation, management and monitoring (Jones 2004). To date key findings have been dominated by the natural sciences and only more recently have included options for adaptation and mitigation. The key findings in earlier reports delineated the scope for uncertainty guidance provided for both the AR4 and AR5 reports. The focus has therefore concentrated on scientific findings describing the behavior of climate and of systems responding to climate change, projections of change and the risks of climate change.
2 Addressing uncertainties in science and risk
Risk in the IPCC guidance is framed as probability and consequence (IPCC 2007; Mastrandrea et al., 2010; Schneider, 2001; Schneider et al. 2007). Risk is most often addressed as a linear process where a risk is identified, analyzed, evaluated and managed, however interacting risks can become very complex. Probability and consequence can apply to event and outcome-based approaches (Sarewitz et al. 2003), where risk can either be characterized as the result of a series of events or as the likelihood of exceeding or attaining a given outcome (Jones and Preston 2011). The international standard on risk defines risk as the effect of uncertainty on objectives (ISO 2009). This broader definition factors in not only the likelihood of a risk occurring, but also whether a set of objectives may be achieved or not when options for risk management themselves are uncertain.
The US Climate Change Science Program, in preparing an extensive survey of methods suitable for a range of different aspects of uncertainty management, warns that simple recipe-based approaches are insufficient (Climate Change Science Program 2009). This highlights the tension between developing the appropriate context to support decision-making in complex situations and communicating uncertainty clearly and simply. A longer-term research program may be needed to develop appropriate material to service the entire scope of IPCC findings.
The uncertainty guidance needs to support scientific findings in an environment where uncertainty is being exploited to undermine climate science and policy (Kitcher 2010; Oreskes and Conway 2010). Two of the key measures recommended here are to strengthen the philosophical basis for uncertainty management as flagged by Kitcher (2010) and to better articulate the relationship between science and risk and between tame and complex risks. The next section addresses the first point and the following section the latter.
2.1 Epistemic and ontological treatment of scientific uncertainty
The confidence-likelihood metrics of the uncertainty guidance form an epistemological-ontological structure: confidence is epistemological and likelihood is ontological. The twinned basis for a key scientific finding combines ontological reasons—what the author team knows—and epistemological reasons—how confident are they in that knowledge—for a particular conclusion or set of conclusions, describing as Dancy (1995) puts it our rights to the beliefs we have. To date, key findings have mainly been communicated ontologically: “This is what the science says”, as interpreted by an expert panel. The epistemic basis of those findings has largely relied on the authority of the IPCC and the comprehensiveness of the assessments, making the findings vulnerable to attack if that authority can be undermined.
Scientific findings on climate change are being interpreted by the broader public through the lens of risk. The science of climate change itself is viewed as harmful by a specific set of interest groups and world views (Oreskes and Conway 2010). Scientific consensus is often confused with group consensus; the public are largely unaware of the role of theory in establishing scientific knowledge, especially confusing straightforward refutation with falsification. The common message conveyed in the media and online is that if one can establish a statistical trend different to a forecast trend, the underpinning theory is falsified. Most of those statistical analyses are easily shown to be invalid (Enting 2007).
Making purely epistemological statements about the knowledge contributing to a finding will assist in the communication of controversial science. This is a different role than using scientific confidence to inform the precision of likelihoods. For example, the existence of a risk can attract high confidence but have low predictability; e.g., future sea level rise from melting of the Greenland Ice Sheet.
The unstructured approach the IPCC has taken to epistemological issues is highlighted by Chapter 6 of (IPCC 2007) Robust findings and key uncertainties, which reads like a shopping list. Structure could have been provided by identifying both positive and negative heuristics grounded in the philosophy of science. This identifies the core theory supporting the underpinning science and describes where that theoretical base is expanding or contracting by documenting changes in explanatory power (Lakatos 1970). The core theory for anthropogenic climate change is the direct radiative forcing response of greenhouse gases and the positive atmospheric temperature feedback in response to radiative forcing. Most of the theory that surrounds this is ancillary or supporting theory. An ancillary hypothesis may attract low confidence, but overturning it does not threaten the basic physics of the greenhouse effect. This falls into the area of science that Kuhn referred to as the “puzzle solving” of normal science (Kuhn 1962; 1970). Uncertainties about how warming trends, changes in sea ice or the global carbon cycle differ from model projections, for example, cannot be used to disprove the core theoretical basis behind the theory of greenhouse gas-induced climate change. All they show is that gaps in scientific knowledge still exist.
Although core climate and ancillary theory, and its supporting evidence are described in the scientific literature and in successive IPCC reports (e.g., EPA 2009; Weart 2008) this process could be formalized by updating the methodical scientific research program proposed by Lakatos (1970) with newer material from authors such as Kitcher (2001) and Roush (2005). Large climate research programs such as those conducted by the World Climate Research Programme and the International Geosphere-Biosphere Programme closely resemble Lakatos’ framework. Methodical falsification of scientific theory can only be achieved if core theory is overturned and an alternative theory with better explanatory power can be proposed (Popper 1972). Core greenhouse theory is supported not only by theory and process models, further evidence from other planets and the behavior of Earth’s palaeoclimate suggesting that no viable alternative exists. Often gaps in ancillary theory are accompanied by very strong empirical evidence. For example, the exact physical processes influencing evaporation and soil moisture are unclear, however; empirical approximations provide robust but imprecise predictions.
Type I and Type II error matrix
Hypothesis (H1) is true
Null hypothesis (H0) is true
Accept hypothesis H1
Type I error
Accept null hypothesis
Type II error
Competing explanations are common in the social sciences. For example, Martin and Richards (1995) describe four different approaches to controversial scientific issues: positivist, group politics, the sociology of scientific knowledge—constructivist and social structural. These approaches can also be used to assess vulnerability and risk. The positivist approach is characterized by rational decision-making models in decision-analytic settings; for example in economics by applying revealed market preference for price and discount rates. The group politics approach brings science to a range of groups in a democratic decision-making process. This approach is commonly in assessing adaptation options where scenarios are used to establish a dialogue between scientists, planners and other stakeholders. Sociological approaches work in two ways: social analysis is applied to scientific knowledge claims as well as to wider social dynamics and all sides in a controversy are examined using the same repertoire of conceptual tools (Martin and Richards 1995). Social structural approaches look at the structure of society, of the state and gender to understand how decision-making manifests within this framework. For example, the same information exercised by a state through its power structures may be very different to that exercised at the community level.
In conclusion, a methodological approach to both epistemological and ontological content that is suitable for both physical and social sciences is recommended. The interpretation (and often communication) of science as risk feeds directly into the risk perceptions of the person who receives that information. The reframing of the science component as a methodical scientific research program with the identification of core and ancillary theory and document changes in explanatory power will help articulate the difference between the science itself and science as risk.
2.2 Tame and complex risks
The relationship between science and risk depends on whether risks can be characterized as being tame or complex, wicked system risks. Tame risks are characterized by a widely agreed conceptual framing of a given risk. Values are bounded and there is an established process for calculating risk and for reconciling calculated and perceived risk. Here, the contribution of scientific information to a risk assessment is relatively straightforward.
Complex risks display a lack of agreement as to how the risks are framed: including the metrics by which risk is measured and disagreements as to whether a given risk or the set of options proposed to solve that risk will cause the greatest harm (van Der Sluijs et al. 2008). These risks are consistent with the characteristics associated with a wicked system (Rittel and Webber 1973): its limits are difficult to constrain, it is multi-causal and has many interdependencies, addressing the problem will have unforeseen consequences, the issue is not stable, has no clear conclusion, is socially complex, is not the responsibility of one organization at single scale, will involve changing behavior and is beset by chronic policy failure (Australian Public Service Commission 2007) and market failure (Garnaut 2008; Stern and Treasury, 2007).
Establish the context: what do we need to take account of and what are our objectives?
Identify the risks: what might happen—how, when and why?
Analyze the risks: what will this mean for our objectives?
Evaluate the risks: which risks need treating and which are the priority for our attention?
Treat the risks: how best should we deal with them?
Although this process can also be followed for complex risks, it is pursued as a non-linear, iterative process. Three important aspects of risk with different epistemic origins are idealized risk, calculated risk and perceived risk. These link the conceptual, technical and believed aspects of risk. Idealized risk is the cognitive model of risk as it is framed during the scoping phase of an assessment. The concept of dangerous anthropogenic influence on the climate system in UNFCCC Article 2 is an example. Calculated risk is the scientific and technical aspect that aims to estimate an idealized risk and its components as accurately as possible. This aspect of risk is the primary focus of the guidance document. Science contributes to calculated risk and to aspects of risk management but risk is explicitly normative due to the pursuit of value-based objectives.
Perceived risk is the rough estimate of risk by a member of the general public (Althaus 2005) and affects how the findings of a risk assessment are communicated, interpreted and acted upon. The heuristics of gain and loss, the diverse cultural mapping of attitudes to risk, different risk-averse and risk-seeking behaviors, rates of time preference and sense of personal identity all affect perceived risk.
The etymology of risk changes throughout the risk assessment process from a noun to a verb as does its meaning (Fillmore and Atkins 1992; Hamilton et al. 2007). When a risk is identified and analyzed, the exposed place, community or process is at risk (noun) because something of value is vulnerable to harm. Under risk evaluation and management, to risk (verb) seeks to gain advantage under uncertainty. This change marks a shift from a state of vulnerability to risk for gain. This transition encounters the uneven heuristics seen in behavioral economics, where the potential for loss is weighted more strongly than the possibility of gain (e.g., Kahneman 2003; Tversky and Kahneman 2000).
Biases affecting risk perception also come from the psychological effects of uncertainty and dread (Weber 2006) that can lead to perceived risk being assessed as up to three orders of magnitude larger than calculated risk (Smil 2008). Familiar and unfamiliar risks can be interpreted similarly, in that familiar risks are often more tolerated than more likely but less familiar risks (Slovic et al., 2004). Culturally-driven perception of risk also leads to the widely varying perception of risks, including those calculated by experts, by different stakeholder groups (Douglas and Wildavsky 1982; Kahan et al. 2007).
For complex risks such as climate change, the different idealised and perceived risks, combined with significant uncertainties in calculated risk, lead to what has been called the risk trap (Beck 2000) or social amplification of risk (Pidgeon et al. 2003) where the risk of climate change clashes with options to manage those risks. For example, if the mitigation of climate is perceived as high risk because of the potential cost to the economy and the difficulty inherent in transforming the global energy-industrial complex, then that risk will be weighed against the risk of climate change itself.
2.3 Distinguishing between science and risk
The main findings of IPCC assessments have been dominated by an approach that links the forecasting paradigm used in scientific assessments to the knowledge gap model or ‘rational’ decision-making framework operating on a predict-then-act basis. This is a useful strategy for tame risks but breaks down in complex policy settings. Furthermore, a rich literature warns against using inflexible top-down policy frames to manage or communicate complex policy situations (e.g., Chaffee 1985; Hart and Banbury 1994; Lindblom 1959; Simon 1976), in favor of a multiple-strategy approach, which is spreading into climate change scholarship (Dessai et al. 2009; Hulme 2009; Kollmuss and Agyeman 2002; Verweij et al. 2006; van Der Sluijs et al. 2008). Scientific paradigms that combine theory, social practice and application (after Masterman 1970) are very different to decision-making paradigms that combine scientific knowledge with other knowledge. A critical-deductive decision-making process that works in a scientific research program will fall flat in a stakeholder-mediated assessment process. The diversity of approaches in managing risk needs to feature in the communication of key findings and the current guidance does not clearly reflect this.
Risk matrix taking account of when the cost of risk mitigation (RM) is uncertain compared to the damages avoided in risk
Risk (R) is true
No risk (R0) is true
If Cost(R)>Cost(RM), then right decision
False positive, ancillary benefits
Delayed action to increase confidence in p(R|R0) or p(Cost RM>Cost R)
If new knowledge decreases Cost(RM) relative to Cost(R), then right decision
Weak false positive, weak ancillary benefits
No action—accept R0
No action—reject RM
The cost or other risk factors associated with risk mitigation or probability of it being successful have nothing to do with the success or not of the underlying scientific hypotheses leading to the assessment of risk. The epistemology of science is different to that of risk, largely because of the explicit value component involved in risk. Furthermore, the types of decisions in Table 2 are accompanied by deep uncertainty requiring a very different treatment to conventional scientific or tame risk problems (Dupuy and Grinbaum 2005; Kandlikar et al. 2005).
When communicating key findings present science with its claims of objectivity separately from risk with its value components and treating the uncertainty for each separately.
Take a more philosophically sound approach to science communication by clearly applying epistemological and ontological structure to the confidence–likelihood construct.
Apply a methodical research program overlay to the science to separate core theory from ancillary theory and make better use of positive and negative heuristics to show where knowledge is progressing or declining.
Be clear about competing explanations, especially in the social sciences.
Be more aware of decision-making paradigms and approaches in framing policy-relevant advice on risks and risk management options.
The current guidance is a step forward but the issue is whether a leap is required. Interdisciplinary paradigms are needed to assess and manage uncertainty and its contribution to decision-making in complex settings. Given the great need for policy-relevant information suitable for a wide range of contexts, a paradigm shift in uncertainty management is required.
The late Steve Schneider pioneered the management and communication of uncertainty and risk in climate change and championed my early work in the Third Assessment Report and in this journal for which I will always be grateful. Professor Peter Sheehan is thanked for ongoing discussions on a range of these issues.
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
- Ad Hoc Working Group on long-term Co-operative Action (2010) Outcome of the work of the ad hoc working group on long-term cooperative action under the convention United Nations Framework Convention on Climate Change, BonnGoogle Scholar
- Australian Public Service Commission (2007) Tackling wicked problems: A public policy perspective. Contemporary government challenges. Australian public service commission. Australian Public Service Commission, CanberraGoogle Scholar
- Beck U (2000) Risk society revisited: theory, politics and research programmes. In: Adam B, Beck U, van Loon J (eds) The risk society and beyond: critical issues for social theory. Sage, LondonGoogle Scholar
- Chaffee EE (1985) Three models of strategy. Acad Manag Rev 10(1):89–98Google Scholar
- Climate Change Science Program (2009) Best practice approaches for characterizing, communicating and incorporating scientific uncertainty in climate decision making. Synthesis and analysis product 52. National Oceanic and Atmospheric Administration, Washington, D.C.Google Scholar
- Committee to Review the Intergovernmental Panel on Climate Change Science Program (ed) (2010) Climate change assessments: Review of the processes and procedures of the ipcc. InterAcademy Council, AmsterdamGoogle Scholar
- Dancy J (1995) Problems of epistemology. In: Honderich T (ed) The oxford companion to philosophy. Oxford University Press, Oxford, pp 245–248Google Scholar
- Dessai S, Hulme M, Lempert RJ, Pielke RJ (2009) Climate prediction: a limit to adaptation? In: Adger N, Goulden M, Lorenzoni I (eds) Living with climate change: are there limits to adaptation? Cambridge University Press, CambridgeGoogle Scholar
- Douglas M, Wildavsky AB (1982) Risk and culture: an essay on the selection of technical and environmental dangers. University of California Press, BerkeleyGoogle Scholar
- Enting IG (2007) Twisted: the distorted mathematics of greenhouse denial. Australian Mathematical Sciences Institute, MelbourneGoogle Scholar
- EPA (2009) Endangerment and cause or contribute findings for greenhouse gases under section 202(a) of the clean air act: EPA’s response to public comments. Environment Protection Authority, Washington DCGoogle Scholar
- Fillmore CJ, Atkins BT (1992) Towards a frame-based organization of the lexicon: the semantics of risk and its neighbors. In: Lehrer A, Kittay E (eds) Frames, fields, and contrasts: new essays in semantics and lexical organization. Lawrence Erlbaum, Mahwah, pp 75–102Google Scholar
- Garnaut R (2008) The Garnaut climate change review: final report. Cambridge University Press, Port MelbourneGoogle Scholar
- Giddens A (2009) The politics of climate change. Polity, CambridgeGoogle Scholar
- Hulme M (2009) Why we disagree about climate change: understanding controversy, inaction and opportunity. Cambridge University Press, CambridgeGoogle Scholar
- IPCC (1998) Principles governing IPCC work. Intergovernmental Panel on Climate Change, GenevaGoogle Scholar
- IPCC (2005) Guidance notes for lead authors of the IPCC fourth assessment report on addressing uncertainties. IPCC, GenevaGoogle Scholar
- IPCC (2007) Climate change 2007: synthesis report. Cambridge University Press, CambridgeGoogle Scholar
- ISO (2009) ISO 31000:2009 risk management—principles and guidelines. International Organisation for Standardisation, GenevaGoogle Scholar
- Jones RN, Preston BL (2011) Adaptation and risk management WIRES. Clim Change. doi: 10.1002/wcc.97
- Kahan DM, Braman D, Slovic P, Gastil J, Cohen GL (2007) The second national risk and culture study: making sense of—and making progress in—the American culture war of fact. SSRN eLibraryGoogle Scholar
- Kuhn TS (1962) The structure of scientific revolutions. University of Chicago Press, ChicagoGoogle Scholar
- Kuhn T (1970) Logic of discovery or psychology of research? In: Lakatos I, Musgrave A (eds) Criticism and the growth of knowledge. Cambridge University Press, Cambridge [Eng.], pp 1–23Google Scholar
- Lakatos I (1970) Falsification and the methodology of science research programmes. In: Lakatos I, Musgrave A (eds) Criticism and the growth of knowledge. Cambridge University Press, Cambridge [Eng.], pp 91–196Google Scholar
- Manning M, Petit M, Easterling D, Murphy J, Patwardhan A, Rogner H-H, Swart R, Yohe G (eds) (2004) IPCC workshop on describing scientific uncertainties in climate change to support analysis of risk and options. IPCC, GenevaGoogle Scholar
- Martin B, Richards E (1995) Scientific knowledge, controversy, and public decision-making. In: Jasanoff S, Markle GE, Petersen JC, Pinch T (eds) Handbook of science and technology studies. Sage, Thousand Oaks, pp 506–526Google Scholar
- Masterman M (1970) The nature of a paradigm. In: Lakatos I, Musgrave A (eds) Criticism and the growth of knowledge. Cambridge University Press, Cambridge [Eng.], pp 59–89Google Scholar
- Mastrandrea MD, Field CB, Stocker TF, Edenhofer O, Ebi KL, Frame DJ, Held H, Kriegler E, Mach KJ, Matschoss PR, Plattner G-K, Yohe GW, Zwiers FW (2010) Guidance note for lead authors of the IPCC fifth assessment report on consistent treatment of uncertainties. Intergovernmental panel on climate change. IPCC, GenevaGoogle Scholar
- Morgan MG, Henrion M, Small M (1990) Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press, CambridgeGoogle Scholar
- Moss R, Schneider S (2000) Uncertainties. In: Pachauri R, Taniguchi T, Tanaka K (eds) Guidance papers on the cross cutting issues of the third assessment report of the IPCC. IPCC, GenevaGoogle Scholar
- National Assessment Synthesis Team (2001) Climate change impacts on the united states: The potential consequences of climate variability and change. Cambridge University Press, Cambridge, UK, and New York, USAGoogle Scholar
- Oreskes N, Conway EM (2010) Merchants of Doubt: how a handful of scientists obscured the truth on issues from tobacco smoke to global warming. Bloomsbury Press, New YorkGoogle Scholar
- Pidgeon N, Kasperson R, Slovic P (eds) (2003) The social amplification of risk. Cambridge University Press, CambridgeGoogle Scholar
- Popper KR (1972) The logic of scientific discovery. 6th impression revised. edn. Hutchinson, LondonGoogle Scholar
- Schneider SH, Semenov S, Patwardhan A, Burton I, Magadza CHD, Oppenheimer M, Pittock AB, Rahman A, Smith JB, Suarez A, Yamin F (2007) Assessing key vulnerabilities and the risk from climate change. In: Parry ML, Canziani OF, Palutikof JP, van der Linden PJ, Hanson CE (eds) Climate change 2007: impacts, adaptation and vulnerability. Contribution of working group ii to the fourth assessment report of the intergovernmental panel on climate change. Cambridge University Press, Cambridge, pp 779–810Google Scholar
- Smil V (2008) Global catastrophes and trends: the next fifty years. MIT Press, CambridgeGoogle Scholar
- Stern NH, Treasury GB (2007) The economics of climate change: the stern review. Cambridge University Press, CambridgeGoogle Scholar
- Tversky A, Kahneman D (2000) Judgement under uncertainty: heuristics and biases. In: Connolly T, Arkes HR, Hammond KR (eds) Judgment and decision making: an interdisciplinary reader. Cambridge University Press, Cambridge, pp 35–52Google Scholar
- Watkiss P (2011) Aggregate economic measures of climate change damages: explaining the differences and implications WIRES. Clim Change. doi: 10.1002/wcc.111
- Weart SR (2008) The discovery of global warming. Harvard University Press, CambridgeGoogle Scholar