Keywords

Introduction

Human activity has become a geologic-scale force, changing landscape and climate at increasing rates in our effort to supply societies growing demand for water, energy and food. A fundamental scientific and societal question of our time is: how will water, energy and biogeochemical cycles be altered by this activity, and when and where will critical thresholds be reached (Gleeson et al. 2020)? This insight, if available at the local or regional scale, is needed to guide decision- and policy-makers in the development of adaptation strategies and in designing infrastructure solutions suitable for future conditions (Barron 2009).

To develop adaptation strategies, impact models are needed to translate climate signals into hydrological, ecological or other decision-relevant variables to understand potential implications of future climate for security issues related to water (droughts and floods), food, energy and health. An important question when using such impact models is how they have been evaluated regarding their ability to perform their task adequately (Wagener et al. 2010)? Most often, the task of addressing this question is referred to as model validation. In a recent review of the validation of resource management models for a wide range of uses including scenario modelling, Eker et al. (2018) found that data-based strategies to model validation still prevail. The use of historical observations, to show that a model can reproduce observed system responses, remains the main approach to demonstrate that a model is a valid representation of reality. The use of the term validation itself has been criticized, because it is overpromising in the sense that it suggests that a model has been established as being true, rather than adequate for the task at hand (Oreskes et al. 1994). Therefore the author uses the term evaluation in this short paper to suggest that we can ever only achieve an incomplete and conditional assessment of a model’s suitability.

Any approach to evaluation suffers from multiple problems. First, the future might be significantly different from the past and demonstrating that a model is a realistic representation of the past system does not necessarily guarantee that it will reflect the future system. Many studies have therefore tried to create some type of resampling of the past to better reflect future conditions during model calibration/evaluation (e.g. Fowler et al. 2018; Singh et al. 2013), though this is only possible within limits. For example, even significant drought periods in the past will not fully reflect the combination of atmospheric, societal, land use and other conditions of the future. In some cases, modellers, therefore, prefer to run their models (especially across large domains) in uncalibrated mode, thus relying on the models’ physical realism. This, however, regularly leaves significant performance gaps between models and observed behaviour—a discrepancy that is typically not propagated into the assessment of future model projections. Second, comparing the model to historical data can ignore the intended use of the model, which one might expect to be the main driver of the evaluation strategy. Klemeš (1986) in his seminal paper introduces multiple ideas for validation strategies in relation to intended model use, e.g. related to modelling land use change. These ideas are still rarely fully implemented. Focusing on fitting historical data is also emphasising model performance, rather than model robustness in the presence of unavoidable uncertainties. And third, comparing the model to historical data might also ignore the manner in which stakeholders gain trust in model predictions, especially related to modelling change (Eker et al. 2018). Stakeholders might, for example, care strongly whether the model structure reflects an understanding of the real-world system that is consistent with their own (Mahmoud et al. 2009).

In this brief paper, how the use of global sensitivity analysis can be beneficial in enabling model evaluation elements that are complementary to assessing the model fit to historical observations would be highlighted (Wagener and Pianosi 2019). It is important to stress that the comparison of historical data is not useful, rather that it is insufficient. So, in this paper, we will briefly discuss using three examples from previously published studies and conclude with some general remarks and suggestions.

Standard Evaluation Questions We Can (and Should) Ask Using Global Sensitivity Analysis

Do the Parameters That Reflect Possible Intervention Levers Adequately Control the Model Output?

A key role of impact models is to create causal links between cause and effect variables, especially in the context of developing adaptation strategies. We might, for example, want to understand how much land use choices such as deforestation/reforestation control the level of downstream flooding under future climate conditions, or we might want to know the level of influence of human activities such as groundwater pumping on the overall drought risk under potential future warming scenarios. We, therefore, have to demonstrate that the parameters reflecting these intervention levers (such as those describing land use or human activities) actually exert an adequate control on the model output consistent with our current understanding.

For example, Butler et al. (2014) performed a comprehensive variance-based sensitivity analysis of a doubled-CO2 stabilization policy scenario generated by the globally aggregated dynamic integrated model of climate and the economy (DICE) (Fig. 5.1). The authors identified dominant processes by quantifying high sensitivities in model parameters relating to climate sensitivity, global participation in abatement and the cost of lower-emission energy sources. More importantly, in the context of this short paper, the authors did not find relevant sensitivities to other parameters such as those related to land use, that one might have expected to exert a stronger influence than the model shows. This result might suggest that certain intervention strategies cannot be assessed using the model in this particular example.

Fig. 5.1
figure 1

Reproduced from Butler et al. 2014)

Results of the study by Butler et al. (2014). a Sensitivities of the net present value of climate damages. The variance decomposition-based results are shown for first-order (filled circles), total-order (hollow rings), and second-order (connecting lines) indices. Diameters of the first- and total-order sensitivity circles are proportional to their respective sensitivity indices. Total sensitivities include first order and all higher-order (parameter interaction) sensitivities. The legend shows the extreme values for these metrics. Sensitivities of <1% are not shown; sensitivities of higher order than 2 are not explicitly shown. b Schematic diagram of the DICE model. Exogenous parameters are in italics. Parameters in bold blue italic are sampled in this study (

Full size image

Are Dominant Uncertainties Changing Along the Projection Timeline?

It is further relevant to understand which uncertainties dominate the model output, especially over long time periods where levels of uncertainty might change considerably. Le Cozannet et al. (2015) used time-varying global sensitivity analysis to determine the factors that most strongly control the vulnerability of coastal flood defences over time (Fig. 5.2). They found that—for their question—global climate change scenarios only matter for long-term planning while local factors such as near-shore coastal bathymetry reflected in the wave setup parameter dominated in the short and mid-term (~over the next 50 years). The authors claim that wave setup uncertainty is often neglected in coastal hazard assessments studies. Global sensitivity analysis reveals that failing to incorporate the uncertainty in this process may invalidate conclusions and may lead to an overestimation of the effects of other drivers at least for short and mid-term planning periods. An assessment of the robustness of the model projections to input uncertainties thus has to consider the time-varying influence of these uncertainties.

Fig. 5.2
figure 2

(Reproduced from Le Cozannet et al. 2015)

The study by Le Cozannet et al. (2015) provides an example of using GSA to support long-term assessments; in this case of coastal defences. The figure shows the temporal sensitivity of predicted coastal defence vulnerability (specifically the output metric is the yearly probability of exceeding the threshold height of coastal defences). The figure shows that dominant drivers change significantly over time; for example global climate change scenario only matters beyond 2070 while offshore extreme values have no influence after that. Interestingly, for the time period up to 2050, the dominant factor is the ‘wave set-up’ parameter, which accounts for sea-level rise induced by wave breaking

Full size image

Are Dominant Modelled Processes Changing with Climate?

And finally, how strongly different modelled environmental processes control the output of adaptation models can vary strongly with climate or other boundary conditions. Models behave differently depending on the climatic boundary conditions they are applied in, regardless of the level of physics the model is based on (Rosero et al. 2010). Figure 5.3 shows some results of a study by van Werkhoven et al. (2008) who tested the sensitivity of a model’s output (streamflow) to the parameters of a lumped rainfall-runoff model across 12 US catchments with very different climatic boundary conditions. The authors found (for both high flow and low flow conditions) that the controlling parameters varied considerably across climatic gradients. They further found that the spatial variability in sensitivity across catchments was similar to that observed within catchments when assessed across wet and dry years. This result suggests that for climate change projections, parameters (processes) that control the model behaviour for the historical period will likely differ from those that control the model output under new climatic boundary conditions. Global sensitivity analysis can provide insight into the degree of such changing model behaviour if a model is tested along climatic gradients.

Fig. 5.3
figure 3

van Werkhoven et al. (2008) showed that the sensitivity of the output of a widely used rainfall-runoff model (SAC-SMA) with respect to its parameters, changes strongly along a climatic gradient (defined by the aridity index: precipitation/potential evapotranspiration (P/PE)). Sensitivity indices (SI) for the model parameters (LZTWM, UFZWM, PFREE, LZFPM) show a strong correlation with the aridity index in relation to both high flows (RMSE) and low flows (TRMSE). This result suggests that dominant modelled processes change along a climatic gradient (Reproduced from van Werkhoven et al. 2008)

Full size image

Conclusions and Recommendations

Validation of impact models—an important task for developing adaptation strategies to climate change and for gaining stakeholder confidence—cannot be based on assessing a model’s fit to historical data alone, even though such assessment can clearly play a role in establishing confidence in a model (Fowler et al. 2018; Eker et al. 2018). It is further important that evaluation strategies are linked to the intended model use (Klemes 1986). In this context, global sensitivity analysis is a valuable tool to complement any data- and performance-based validation strategy since it allows us to make the model and its simulations significantly more transparent. It is important to stress here that sensitivity analysis can be applied regardless of whether observations of the system response are available or not, thus making it very suitable for understanding the behaviour of models under potential future conditions (Wagener and Pianosi 2019). Possible questions for sensitivity analysis are:

Do the model parameters that are linked to potential adaptation strategies (e.g. via land use choices) exert expected levels of control on the modelled output?

Which uncertainties are likely to dominate the model output during the relevant assessment period?

Which modelled processes are likely to dominate the model output under the projected climatic conditions (rather than under the conditions for which historical observations are available) and are we confident in those estimated parameters?

Are model projections (and more importantly subsequent decisions) robust to input uncertainties?

How sensitive are the model projections to model assumptions?

Global sensitivity analysis can provide a valuable additional component to strengthen our confidence as well as the confidence of stakeholders in climate change impact models (Wagener and Pianosi 2019; Saltelli et al. 2020). Recent studies have further demonstrated that such sensitivity analysis can be performed even on highly complex models (Maples et al. 2020) or on those covering a global domain (Reinecke et al. 2019).