Introduction to Assessing climate models: knowledge, values and policy
1 The philosophy of climate modelling and the roles of climate models workshop
Climate models have been of substantial societal significance for several decades now, as they are a central tool relied upon in the study of global climate change. They have not, however, been the focus of much philosophy of science. This has begun to change in recent years, with an emerging literature on philosophical issues in climate modelling. The questions addressed in this literature are diverse, but in broad terms much of the work is concerned with the evidential roles of climate models, i.e., with what they can (and cannot) contribute when it comes to evaluating hypotheses about past, present and future climate change. In particular, philosophical attention has tended to focus on the assessment of climate models and their predictions, especially questions related to confirmation (e.g., Lloyd 2009, 2010; Parker 2009; Lenhard and Winsberg 2010; Steele and Werndl 2013; Katzav 2013, 2014) and to uncertainty (e.g., Petersen 2012; Parker 2010; Biddle and Winsberg 2010; Winsberg 2012; Katzav et al. 2012; Frigg et al. 2013).
In October 2013, a workshop was held in Eindhoven, The Netherlands, entitled “The Roles of Climate Models: Epistemic, Ethical and Socio-Political Perspectives.” One core aim of the workshop was to advance understanding of the evidential roles of climate models, but another was to reflect on their many other roles: in framing policy discussions, in integrating knowledge across disciplines, in shaping the organization and direction of climate science research, and so on. To this end, invited participants included climate scientists, philosophers, sociologists, historians and policy makers.
Ultimately, those two workshop aims were only partially realized: much of the discussion remained focused on the evidential roles of climate models and in particular on climate model assessment, albeit with some attention to value and policy dimensions of climate modelling. This is reflected in the papers that comprise this special issue, an issue that grew out of the workshop: the papers by Mathias Frisch and Gregor Betz contribute to existing threads in the philosophical literature on climate model assessment; the paper by Gavin Schmidt and Steven Sherwood provides an insider’s perspective on climate model development and assessment; the paper by Kristin Intemann considers how non-epistemic values might legitimately influence climate modelling practice; and the paper by Liam Heaphy discusses how the aim of informing decision making can impact the design of climate modelling studies and should also figure in their evaluation. We summarise and discuss these papers in more detail below.
The workshop was also motivated by the recognition that there is substantial diversity in current views regarding the evidential roles that climate models can and should play, e.g., whether they should play a primary role in establishing what climate sensitivity is, what they can tell us about future climate change or the causes of recent climate change, etc. We suspect that this diversity stems, at least to some extent, from differences in underlying epistemic principles and assumptions; scientists’ conclusions are clearly informed by theory, empirical evidence and modelling, but they also are shaped by views of what can count as good scientific evidence, where the burden of proof rests and so on. Such epistemic principles and assumptions are often contestable, and debates about them are squarely within the purview of philosophy of science. This suggests that there may be valuable work to be done by philosophers and others here – in unpacking and scrutinizing the epistemic assumptions that undergird both mainstream and alternative views of the evidential value of climate models and, indeed, of evidence in climate science more broadly.
Accordingly, a third aim of the workshop was to encourage discussion among individuals with diverse views on whether climate models are suitable for the various epistemic and policy purposes to which they are put. We aimed to include voices representing the dominant view that climate models are generally suitable for these purposes as well as dissenting voices. The workshop was a success in this regard, with, for example, some of the attending climate scientists strongly endorsing and some strongly objecting to the dominant view in the course of workshop presentations and discussions. The papers in the current collection, however, do not fully reflect this diversity: challenges for the dominant view are posed, especially in Betz’s paper, but scientists who object to the dominant view are not represented.
2 Model development and assessment: a climate scientist’s perspective
Schmidt and Sherwood’s article nicely realises the goal of interdisciplinary communication about climate models. The authors are prominent climate scientists who provide us with an insider’s perspective on the development and assessment of General Circulation Models (GCMs). They also engage with recent work in the philosophy of modelling.
Schmidt and Sherwood observe that GCM development consists in improving model efficiency, accuracy, scope and realism; but much of their discussion of this development focuses on tuning. They note that GCMs include a large number of loosely-constrained parameters. After GCM construction is complete, a small number of these parameters are tuned so as to minimise errors in simulations of a number of important emergent climatic properties, such as the top-of-the-atmosphere radiative imbalance. According to Schmidt and Sherwood, this provides only a limited ability to improve overall simulation skill; tuning to improve simulation of one property of climate will usually degrade simulation of others. In addition, they suggest that, insofar as non-epistemic values play a role in choosing what to tune for, these are aesthetic or practical rather than the kinds of social and ethical values that Eric Winsberg (2012) and Heather Douglas (2000) have argued must influence uncertain methodological choices in science.
In their discussion of model evaluation, Schmidt and Sherwood emphasise the need for tests of model predictions against data to which models have not been explicitly tuned. In this context, they list a number of successful out-of-sample model predictions, while recognising the need to balance these successes against cases in which out-of-sample predictions have failed. They also elaborate on the role of multi-model ensembles in evaluating models, noting that experts often assign ensemble simulation results various levels of confidence in light of available evidence. In their view, this is effectively an informal sort of Bayesian updating – one that does not involve explicit calculation of likelihoods or posterior probabilities.
Finally, Schmidt and Sherwood look at a variety of philosophical perspectives on scientific modelling. Here they engage critically with Paul Humphreys’ (2009) arguments that simulation introduces new philosophical problems, including his suggestion that humans and computers share epistemic authority when it comes to complex simulations.
3 Frisch on why use-novelty matters in climate model assessment
Frisch’s contribution delves even more deeply into the issue of climate model tuning, in particular the issue whether data used in tuning a climate model can also confirm the model (or the model’s adequacy or skilfulness). His paper is a reply to a recent article by Steele and Werndl (2013), which argues that, from a Bayesian perspective, there is no problem with using the same data to tune and confirm a climate model; for the Bayesian, tuning (or ‘calibration’) “is simply the common practice of testing hypotheses against evidence” (ibid, p.615). Frisch aims to challenge Steele and Werndl’s Bayesian argument on its own terms. Moreover, recognizing that successful tuning is a special case of accommodation, he aims to defend an alternative view of the significance of prediction versus accommodation in modeling.
In response to Steele and Werndl’s Bayesian argument, Frisch reminds us that there isn’t just a single Bayesian answer to the question whether prediction possesses an epistemic advantage over accommodation. Instead, the answer depends on what one thinks should be included in the background information that is conditionalized on when performing Bayesian updating; different Bayesians can take different views on this, as attempts to resolve the problem of old evidence have made clear.
Regarding prediction versus accommodation, he defends a “moderate predictivism”: in some circumstances, successful use-novel prediction provides stronger evidence of a model’s reliability or skillfulness than successful accommodation of data. Among these circumstances are those in which the model is ‘epistemically opaque’ in the sense that we do not understand its detailed workings nor whether it adequately represents the mechanisms underlying phenomena of interest. In such situations, successful use-novel prediction provides stronger evidence that the model has ‘latched on to’ the mechanisms underlying a phenomenon of interest and thus that it will be reliable when it comes to answering some range of further questions about the modeled system; with accommodation achieved via tuning, there is a greater worry that model-data fit is achieved despite significant (compensating) errors in the model’s representation of underlying mechanisms. Frisch argues that climate models are epistemically opaque and thus that use-novel prediction carries more epistemic weight than accommodation in this context.
Frisch’s analysis is nicely complementary to that of Schmidt and Sherwood. The latter accuse Steele and Werndl of ignoring “the fact that results that are predicted “out-of-sample” demonstrate more useful [model] skill than results that are tuned for (or accommodated)”. Frisch in effect offers a justification for this predictivist sentiment in the climate context.
4 Betz’s modal epistemology
Betz rejects the kind of probabilistic approach to assessing climate model predictions advocated by Schmidt and Sherwood, and endorsed by many climate scientists. An alternative proposal (Stainforth et al. 2007) is that climate models can be used in possibilistic prediction, that is, that they might be used in showing that particular outcomes are serious possibilities. Possibilistic prediction, however, also faces challenges. Betz defines serious possibilities as possibilities that are consistent with background knowledge. This raises the question of how, given that climate models are known to include false assumptions and thus are inconsistent with background knowledge, climate model simulations can be used to show that some predictions are serious possibilities.
The literature includes a number of responses to Betz’s challenge. One response, adopted by Katzav (2014), is to dissolve the challenge by denying that being consistent with background knowledge is either necessary or sufficient for being a serious possibility. Betz (2010) previously responded to his own challenge by arguing that, although climate models cannot support possibilistic predictions, they can help us articulate or think about them for the first time. In his present contribution, Betz continues to hold that serious possibilities are those that are consistent with background knowledge while reconsidering whether climate models can be used to show that certain predictions are serious possibilities thus construed.
Betz’s idea is that, although climate models are inconsistent with background knowledge, some of their predictions might have the property of matching those that would result from climate models that are consistent with background knowledge. If we manage to show that some predictions of actual models do have this matching property, we will thereby have shown that the predictions are serious possibilities. Betz’s paper examines the kinds of arguments that would be needed in order to show that some of the predictions of actual climate models match those of climate models that are consistent with background knowledge. He argues that there are serious challenges that need to be overcome before such arguments can be sufficiently developed, while leaving open the question whether these challenges can indeed be overcome.
5 Intemann: models as bearers of values
Intemann’s contribution is concerned with the roles of non-epistemic values in climate modelling. Recent discussion of this issue has tended to focus on whether social and ethical values in fact influence methodological choices in climate modelling, whether such influence is unavoidable and whether it should be avoided if possible (see Winsberg 2012; Betz 2013; Parker 2014). As noted above, Schmidt and Sherwood acknowledge that aesthetic and pragmatic values influence methodological choices in climate modelling, but they seem to think that social and ethical values do not and should not. Intemann is concerned with whether social and ethical values should influence these choices – she argues that they should – but her primary focus is on when they should do so.
Her arguments are grounded in a view of the nature of the aims of scientific research. In particular, she rejects the idea that traditional epistemic aims of research can be easily isolated or distinguished from social and policy aims. We don’t want to know just anything about future climate; we want to know things that will facilitate policy decisions that protect the things we care about. If the aims of research are epistemic and social, and if methodological decisions in science are justified by the aims of research, then social aims – alongside and intertwined with epistemic aims – can have a legitimate role to play in shaping methodological decisions in science. The question of when the influence of social and ethical values is legitimate is then answered by what Intemann calls the aims approach: the influence is legitimate when it promotes “democratically-endorsed” epistemic and social aims of research, i.e., aims that are supported by a broad range of stakeholders. Crucially, these are not aims chosen by individual scientists or even groups of scientists.
Intemann illustrates that the aims approach captures intuitions about when epistemic and social values legitimately influence methodological decisions in science. For instance, intuitively, a desire to know about worst-case scenarios – because we consider it important to protect against them – can justify different decisions in model development than a desire to know about most-likely outcomes. Likewise, since in practice simulating one climate variable more accurately often means simulating another less accurately, choices will have to be made about which variables should take priority (see also Winsberg 2012); intuitively, these choices are appropriately influenced by what we consider more important – from a social or ethical point of view – to simulate accurately. Once again, the relevant “we” in these examples are stakeholders, and the aims that can legitimately shape these choices are democratically endorsed ones.
A remaining challenge, which Intemann readily acknowledges, is to give a more complete account of what constitutes a “democratically endorsed” epistemic or social aim of research. As a first step, she notes recent efforts to incorporate stakeholders at various points in research and assessment processes in the study of climate change, including efforts made by the UK Climate Impacts Program, the Intergovernmental Panel on Climate Change and others.
6 Heaphy: models in the service of policy
Efforts to incorporate stakeholders are among the activities discussed by Heaphy in his detailed examination of one of the projects completed by the UK Climate Impacts Program: the UK Climate Projections 2009 (UKCP09). This complex and sophisticated modelling study was undertaken with the aim of providing probabilistic projections of future climate change for the UK at regional and local scales, in order to facilitate adaptation planning and decision making by government and industry across a wide range of sectors.
UKCP09 has previously attracted the attention of philosophers of science, who have expressed doubts that assigning (what appear to be) precise probabilities to regional changes in climate is justified, given known limitations of climate models (see e.g., Frigg et al. 2013; Parker 2014). Heaphy’s focus is somewhat different: approaching UKCP09 from a social science perspective, he explores the rationale for producing probabilistic projections in UKCP09, the involvement of users and stakeholders in the UKCP09 process, and the subsequent usability of the projections.
Heaphy explains that UKCP09 was motivated in part by a desire to facilitate risk-based decision making. Previous efforts to provide information about future climate change in the UK had offered scenarios of what future climate might be like without assigning probabilities to different outcomes or otherwise quantifying uncertainties. (This is akin to Betz’s possibilistic approach.) UKCP09, by contrast, did quantify uncertainty about future climate change, in probabilistic terms. Heaphy notes that, although such information was meant to facilitate decision making, and in some cases did so successfully, in other cases users were unable to understand and interpret the probabilistic projections and/or found that decision making was made more difficult with the added information about uncertainties. Moreover, for some decisions, such as those related to design of the built environment under a precautionary attitude, it may not be worth producing detailed probabilistic projections of climate change; approaches that are less resource-intensive may be sufficient.
Emerging from Heaphy’s case study is an important, more general point about the assessment of models and modelling studies: especially when modelling studies are serving joint epistemic and social/policy aims, assessment should consider not just how faithfully models represent their target systems, but also whether the results produced take a form that is relevant and understandable to users. In other words, the adequacy of a model or modelling study can be user-dependent.
7 Future work in the philosophy of climate modelling
The contributions to this special issue help to advance understanding of climate model assessment in a number of ways. Inevitably, however, a number of questions and puzzles remain. Here we identify some that we believe merit additional attention.
Frisch, Schmidt and Sherwood agree that accommodation achieved through explicit tuning provides weaker confirmation of a model’s skilfulness than use-novel prediction. But sometimes known data guide model construction in other ways, e.g., by influencing the form of modelling equations or the decision to use one parameterization rather than another. Schmidt and Sherwood do not raise worries about these other cases of accommodation. Frisch’s moderate predictivism, by contrast, appears to entail that fit to data that guided model construction (even if not used in explicit tuning) provides weaker confirmation than truly use-novel prediction. This would suggest that the significance of climate models’ successes in simulating past and present climate may be weaker and less clear than is suggested by Schmidt and Sherwood’s discussion: weaker, because some of those observational data no doubt did influence choices in model construction; less clear, because exactly what sort of influence occurred is not reported and indeed perhaps not even something that modellers were aware of. The extent to which this is a serious problem requires further exploration.
There are also questions about what use-novel prediction does give us in the climate context. Frisch notes that, even when climate modelling results fit data that were not used in model construction, this may not provide strong evidence of a climate model’s reliability for projecting future changes in climate. One reason is that parts of today’s climate models have been tuned to 20th century data, while the future boundary conditions of interest (e.g., levels of greenhouse gas concentrations) are substantially different from those of the 20th century; given that most use-novel predictions so far are of 20th century climatic conditions, it is difficult to determine their significance. In addition, Schmidt and Sherwood’s observation that only a small subset of parameters are typically employed in tuning climate models suggests that tuning could have been pursued via different routes (e.g., via different parameters), and it is not clear whether the resulting climate models would have performed just as well in simulating 20th century conditions while giving different long-term projections.
Life is not necessarily easier if one gives up on using climate models to determine how climate would actually change under different emission scenarios and settles instead for learning about possibilities. Betz’s argument suggests that at present we can’t even be confident that today’s climate models show us serious possibilities for future climate. This would seem to leave us in a rather bleak place epistemically: the basics of the enhanced greenhouse effect remain clear, but what more can be said about future climate change? Fortunately, pessimism about using models to learn about possibilities seems premature, even if we cannot implement the strategies that Betz identifies for arguing that climate modelling results are consistent with background knowledge. As noted above, some philosophers take the view that being consistent with background knowledge is not necessary for being a serious possibility (see Katzav 2014). Betz’s paper points to the need for a more extended discussion of the conditions under which possibilities should be taken seriously.
Intemann’s arguments and Heaphy’s analysis both point to a need for greater attention to social and policy aims of research when constructing and evaluating climate models. In our experience, and in line with the remarks of Schmidt and Sherwood, this is not a widely-shared view in the climate modelling community. Is this because many practitioners believe that the epistemic and social/policy aims of research can be more easily separated than Intemann claims? If so, perhaps a closer look at the extent to which they are separable, and at the conditions that allow their separation, would be salutary. If, building on Heaphy, changes in modelling practice are called for – changes that make climate modelling more responsive to the social and policy aims of research – it would be good to consider which changes are most urgent and/or most feasible.
Finally, we would like to call attention to some of the topics that we wish would have received more attention at the Roles of Climate Models workshop. As noted above, the discussion at the workshop tended to remain focused on the evidential roles of climate models while there are many other roles that climate models play. They often play a major role in framing policy discussions. They serve as sites of knowledge integration, bringing together information and expertise from a wide range of disciplines. They shape the organization and direction of climate science research in significant ways. More work is needed in philosophy of science and beyond to better understand these and other important roles of climate models.
- Betz, G. (2010). What’s the worst case? The methodology of possibilistic prediction. Analyse und Kritik, 32(1), 87–106.Google Scholar
- Betz, G. (2013). In defence of the value free ideal. European Journal for Philosophy of Science, 3(2), 207–220.Google Scholar
- Biddle, J., & Winsberg, E. (2010). Value judgements and the estimation of uncertainty in climate modeling. In P. D. Magnus & J. Busch (Eds.), New waves in philosophy of science. Basingstoke: Palgrave.Google Scholar
- Katzav, J. (2014) The epistemology of climate models and some of its implications for climate science and the philosophy of science. Studies In History and Philosophy of Modern Physics, 46, 228–238.Google Scholar
- Parker, W. S. (2009). Confirmation and adequacy-for-purpose in climate modelling. Proceedings of the Aristotelian Society, Supplementary Volume, 83(1), 233–249.Google Scholar
- Parker, W. S. (2010). Predicting weather and climate: uncertainty, ensembles and probability. Studies in History and Philosophy of Modern Physics, 41(3), 263–272.Google Scholar
- Steele, K., & Werndl, C. (2013). Climate models, calibration, and confirmation. British Journal for the Philosophy of Science, 64(3), 609–635.Google Scholar
- Winsberg, E. (2012). Values and uncertainties in the predictions of global climate models. Kennedy Institute of Ethics Journal, 22(2), 111–137.Google Scholar