Abstract
We demonstrate and validate a Bayesian approach to model calibration applicable to computationally expensive General Circulation Models (GCMs) that includes a posterior estimate of the intrinsic structural error of the model. Bayesian artificial neural networks (BANNs) are trained with output from a GCM and used as emulators of the full model to allow a computationally efficient Markov Chain Monte Carlo (MCMC) sampling of the posterior for the GCM parameters calibrated against seasonal climatologies of temperature, pressure, and humidity. We validate the methodology by calibrating to targets produced by a model run with added noise. We then demonstrate a calibration of five GCM parameters against an observational data set. The approach accounts for both parametric and structural uncertainties of the model as well as uncertainties associated with the observational calibration data. This enables the generation of statistically rigorous probabilistic forecasts for future climate states. All calibration experiments are performed with emulators trained using a maximum of one hundred model runs, in accord with typical resource restrictions imposed by computationally expensive models. We conclude by summarizing remaining issues to address in order to create a complete and validated operational methodology for objective calibration of computationally expensive models.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
To emphasize this, some writers express such probability distributions in terms of \(P(\cdot {\vert} I)\), where the “I” represents the information available to the individual; e.g., Jaynes (2003).
The expression \(P({\user2{z}} {\vert} \varvec{\theta}) = P({\user2{z}} {\vert} {\user2{f}}, \varvec{\theta})\) results from f being a deterministic function of \(\varvec{\theta}\). However, it does not necessarily follow that \(P({\user2{z}} {\vert} {\user2{f}}, \varvec{\theta}) = P({\user2{z}} {\vert} {\user2{f}})\) unless the relationship between \(\varvec{\theta}\) and f is one to one.
This is if the resulting model output space makes such calculations appropriate; c.f. Sivia and Skilling (2006).
The symbol∼reads as “distributed as”; i.e., samples are concentrated according to the distribution P.
Available at http://www.cs.toronto.edu/~radford/fbm.software.html, (Neal 1996).
It is commonly pointed out that ANNs can have very limited applicability when the goal is to describe the mechanisms behind an observed relationship; e.g., Sanderson et al. (2008).
Typically N, the number of times this process is reiterated, is determined by time and computational resources. Ideally this loop would repeat until a selected convergence criteria is met
The “post burn-in” portion of the sample are the samples that occur after the MCMC chain has converged.
PCA is also referred to as Empirical Orthogonal Function Analysis.
Here we use the term “default” to refer to the values assigned in the original source code for the Planet Simulator model.
While conceptually simple, it has been argued that uniform priors would rarely truly represent ones prior beliefs; c.f., Rougier (2007).
The symbol \(\triangleq\) reads as “equal by definition” and \(N({\user2{z}}{\vert}\cdot)\) represents a Gaussian distribution for z.
By working with the logarithm of the standard deviation, we essentially consider the probability of σ/2 to be equal to the probability of 2σ. This type of treatment is appropriate when describing ‘scale’ parameters whose uncertainties are relative rather than absolute Sivia and Skilling (2006).
Here the discrepancy terms are written \(\varvec{\sigma}_M^2(\varvec{\theta}_i ;{\user2{z}})\) as a reminder that they are sampled jointly with \(\varvec{\theta}_i\) from the posterior, and so are distinct for each parameter set.
A more prudent approach is to perform a cross-validation, where a different element of training data is reserved each time and used to test the predictive ability of the resulting BANN. While this does require additional computing resources, it requires less than running a full GCM only to find out that it was calibrated based on the predictions of a poorly performing emulator. Alternatively, the Bayesian structure of the emulator does allow for more sophisticated methods of model selection, including comparison of Bayes factors or use of the Bayesian information criterion (Lee 2004).
The logarithm is used to avoid computational round off errors.
Note that this sub ensemble is not actually produced by the posterior distribution addressed in Fig. 7 which would require generating a new set of runs with the GCM. Given the limited evolution between the final iterations of the calibration routine for Ensemble A, we assume that this sample is an adequate approximation to that which would be generated by the final posterior.
References
Annan JD, Hargreaves JC (2007) Efficient estimation and ensemble generation in climate modelling. Philos Trans R Soc Lond 365:2077–2088
Craig P, Goldstein M, Rougier J, Seheult A (2001) Bayesian forecasting for complex systems using computer simulators. J Amer Stat Assn 96(454):717–729
Edwards N, Cameron D, Rougier J (2010) Precalibrating an intermediate complexity climate model. Clim Dyn 1–14
Goldstein M, Rougier J (2010) Reified Bayesian modelling and inference for physical systems. J Stat Plan Infer 139
Grosso A, Jamali A, Locatelli M (2008) Iterated local search approaches to maximin Latin hypercube designs. Innov Advan Tech Sys Comp Sci Softw Eng
Holden PB, Edwards NR, Oliver KIC, Lenton TM, Wilkinson RD (2010) A probabilistic calibration of climate sensitivity and terrestrial carbon change in genie-1. Cli Dyn 35(5)
Jackson C (2009) Use of Bayesian inference and data to improve simulations of multi-physics climate phenomena. J Phys 180
Jackson C, Sen MK, Stoffa PL (2004) An efficient stochastic Bayesian approach to optimal parameter and uncertainty estimation for climate model predictions. J Clim 17
Jackson C, Sen M, Huerta G, Deng Y, Bowman K (2008) Error reduction and convergence in climate perdiction. J Clim 21
Jaynes ET (2003) Probability theory; the logic of science, 1st edn. University Press, Cambridge
Kalnay E, Kanamitsu M, Kistler R, Collins W, Deaven D, Gandin L, Iredell M, Saha S, White G, Woollen J, Zhu YMC, Ebisuzaki W, Higgins W, Janowiak J, Mo K, Ropelewski C, Wang J, Leetmaa A, Reynolds R, Jenne R, Joseph D (1996) The NCEP/NCAR 40-year reanalysis project. Bull Amer Meteor Soc 77
Khu ST, Micha G (2003) Reduction of Monte-Carlo simulation runs for uncertainty estimation in hydrological modelling. Hydro Earth Sys Sci 7(5)
Knutti R, Stocker T, Plattner G (2003) Probabilistic climate change projections using neural networks. Clim Dyn 21
Lee H (2004) Bayesian nonparametrics via neural networks. ASA and SIAM
Lunkeit F, Böttinger M, Fraedrich K, Jansen H, Kirk E, Kleidon A, Luksch U (2007a) Planet simulator reference manual, 15 edn. University of Hamburg, Hamburg
Lunkeit F, Blessing S, Fraerich K, Jansen H, Kirk E, Luksch U, Sielmann F (2007b) Planet simulator user’s guide, 15 edn. University of Hamburg, Hamburg
MacKay D (2003) Information theory, inference, and learning algorithms. Cambridge University Press, Cambridge
Mosegaard K, Sambridge M (2002) Monte Carlo analysis of inverse problems. Inv Prob 18
Müller P, von Storch H (2004) Computer modelling in atmospheric and oceanic science, building knowledge. Springer, Berlin
Murphy J, Booth B, Collins M, Harris G, Sexton D, Webb M (2007) A methodology for probabilistic predictions of regional climate change from perturbed physics ensembles. Philos Trans R Soc Lond 365:1993–2028
Nakicenovic N, Alcamo J, Davis G, de Vries B, Fenhann J, Gaffin S, Gregory K, Grbler A, Jung T, Kram T, La Rovere E, Michaelis L, Mori S, Morita T, Pepper W, Pitcher H, Price L, Riahi K, Roehrl A, Rogner HH, Sankovski A, Schlesinger M, Shukla P, Smith S, Swart R, van Rooijen S, Victor N, Dadi Z (2000) IPCC special report on emissions scenarios. Nakicenovic N, Sward R (eds), Cambridge University Press, Cambridge
Neal R (1996) Bayesian learning for neural networks. Springer, New York
Neal R (2003) Slice sampling. Ann Stat 31(3):705–767
Preisendorfer R (1988) Principal component analysis in meteorology and oceanography. Elsevier, Amsterdam
Reynolds R, Rayner N, Smith D, Wang W (2002) An improved in situ and satellite sst analysis for climate. J Clim 15:1609–1625
Rougier J (2007) Probabilistic inference for future climate using an ensemble of climate model evaluations. Clim Change 81:247–264
Rougier J (2008) Efficient emulators for multivariate deterministic functions. J Comp Graph Stat 17(4):827–843
Rougier J, Guillas S, Maute A, Richmund A (2007) Emulating the thermosphere-ionosphere electrodynamics general circulation model. Technical report, statistical and applied mathematical sciences instituted, Research Triangle Park
Sambridge M, Mosegaard K (2002) Monte carlo methods in geophysical inverse problems. Rev Geophys 40(3):1–29
Sanderson B, Piani C, Ingram W, Stone D, Allen M (2008) Towards constraining climate sensitivity by linear analysis of feedback patterns in thousands of perturbed-physics GCM simulations. Clim Dyn 30:175–190
Sivia D, Skilling J (2006) Data analysis a bayesian tutorial, 2nd edn. Oxford University Press Inc., Oxford
Solomon S, Qin D, Manning M, Chen Z, Marquis M, Averyt K, Tignor M, Miller H (2007) Contribution of working group 1 to the fourth assesment report of the intergovermental panel on climate change. Cambridge University Press, Cambridge
Tans P (2009) Mauna Loa CO2 annual mean data. http://www.esrl.noaa.gov/gmd/ccgg/trends/. NOAA/ESRL
Tarasov L, Peltier W (2005) Arctic freshwater forcing of the Younger Dryas cold reversal. Nature 435:662–665
Urban N, Fricker T (2010) A comparison of Latin hypercube and grid ensemble designs for the multivariate emulation of a climate model. Comp Geos 36:746
Villagraon A, Huerta G, Jackson C, Sen M (2008) Computational methods for parameter estimation in climate models. Bayes Analy 3(4):823–850
Wagener T, Boyle D, Less M, Wheater H, Gupta H, Sorooshian S (2001) A framework for development and application of hydrological models. Hydro EarthSyst Sci 5(1):13–26
Acknowledgments
Support provided by: CFI, NSERC, and ACEnet. This work strongly benefitted from discussions with Jonathan Rougier, Radford Neal, Michael Goldstein, Entcho Demirov, and Robert Briggs, as well as from detailed comments provided by two anonymous reviewers. This work also benefitted from numerous discussions at the fall 2010 Mathematical and Statistical Approaches to Climate Modelling and Prediction program hosted by the Isaac Newton Institute for Mathematical Sciences.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Hauser, T., Keats, A. & Tarasov, L. Artificial neural network assisted Bayesian calibration of climate models. Clim Dyn 39, 137–154 (2012). https://doi.org/10.1007/s00382-011-1168-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00382-011-1168-0