Abstract
When making predictions with complex simulators it can be important to quantify the various sources of uncertainty. Errors in the structural specification of the simulator, for example due to missing processes or incorrect mathematical specification, can be a major source of uncertainty, but are often ignored. We introduce a methodology for inferring the discrepancy between the simulator and the system in discrete-time dynamical simulators. We assume a structural form for the discrepancy function, and show how to infer the maximum-likelihood parameter estimates using a particle filter embedded within a Monte Carlo expectation maximization (MCEM) algorithm. We illustrate the method on a conceptual rainfall-runoff simulator (logSPM) used to model the Abercrombie catchment in Australia. We assess the simulator and discrepancy model on the basis of their predictive performance using proper scoring rules. This article has supplementary material online.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Beven, K. (2006), “A Manifesto for the Equifinality Thesis,” Journal of Hydrology, 320, 18–36.
Box, G. E. P., and Cox, D. R. (1964), “An Analysis of Transformations,” Journal of the Royal Statistical Society. Series B. Methodological, 26, 211–252.
Caffo, B. S., Jank, W., and Jones, G. L. (2005), “Ascent-Based Monte Carlo Expectation-Maximization,” Journal of the Royal Statistical Society. Series B. Methodological, 67 (2), 235–251.
Conti, S., Gosling, J., Oakley, J., and O’Hagan, A. (2009), “Gaussian Process Emulation of Dynamic Computer Codes,” Biometrika, 96 (34), 663–676.
Crucifix, M., and Rougier, J. (2009), “On the Use of Simple Dynamical Systems for Climate Predictions,” European Physical Journal, Special Topics, 174, 11–31.
Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977), “Maximum Likelihood from Incomplete Data Via the EM Algorithm,” Journal of the Royal Statistical Society. Series B. Methodological, 39, 1–38.
Doucet, A., de Freitas, N., and Gordon, N. (2001), Sequential Monte Carlo Methods in Practice, Berlin: Springer.
Gneiting, T., and Raftery, A. E. (2007), “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102 (477), 359–378.
Godsill, S., Doucet, A., and West, M. (2004), “Monte Carlo Smoothing for Nonlinear Time Series,” Journal of the American Statistical Association, 99 (465), 156–168.
Goldstein, M., and Rougier, J. (2009), “Reified Bayesian Modelling and Inference for Physical Systems” (with discussion), Journal of Statistical Planning and Inference, 139, 1221–1239.
Gordon, N. J., Salmond, D. J., and Smith, A. F. M. (1993), “Novel Approach to Nonlinear/Non-Gaussian Bayesian State Estimation,” IEE Proceedings. Part F. Radar and Signal Processing, 140, 107–113.
Griffith, A. K., and Nichols, N. K. (2000), “Adjoint Techniques in Data Assimilation for Estimating Model Error,” Flow, Turbulence and Combustion, 65, 469–488.
Higdon, D., Gattiker, J., Williams, B., and Rightley, M. (2008), “Computer Model Calibration Using High-Dimensional Output,” Journal of the American Statistical Association, 103, 570–583.
House, L., Goldstein, M., and Rougier, J. (2008), “Assessing Model Discrepancy Using a Multi-model Ensemble,” MUCM Technical Report 08/07, Durham University and University of Bristol.
Jolliffe, I. T., and Stephenson, D. B. (2003), Forecast Verification: A Practitioner’s Guide in Atmospheric Science, Chichester: Wiley and Sons.
Kavetski, D., Kuczera, G., and Franks, S. W. (2003), “Semi-distributed Hydrological Modelling: a ‘Saturation Path’ Perspective on TOPMODEL and VIC,” Water Resources Research, 39, 1246–1253.
Kennedy, M. C., and O’Hagan, A. (2001), “A Bayesian Calibration of Computer Models” (with discussion), Journal of the Royal Statistical Society. Series B. Methodological, 63, 425–464.
Kuczera, G., Kavetski, D., Franks, S., and Thyer, M. (2006), “Towards a Bayesian Total Error Analysis of Conceptual Rainfall-Runoff Models: Characterising Model Error Using Storm-Dependent Parameters,” Journal of Hydrology, 331, 161–177.
Meng, X. L., and Rubin, D. B. (1991), “Using EM to Obtain Asymptotic Variance-Covariance Matrices: The SEM Algorithm,” Journal of the American Statistical Association, 86, 899–909.
Nash, J. E., and Sutcliffe, J. V. (1970), “River Flow Forecasting Through Conceptual Models Part I—A Discussion of Principles,” Journal of Hydrology, 10, 282–290.
Oakley, J. E., and O’Hagan, A. (2002), “Bayesian Inference for the Uncertainty Distribution of Computer Model Outputs,” Biometrika, 89, 769–784.
Oberkampf, W. L., and Trucano, T. G. (2008), “Verification and Validation Benchmarks,” Nuclear Engineering and Design, 238, 716–743.
Reichert, P., and Mieleitner, J. (2009), “Analyzing Input and Structural Uncertainty of Nonlinear Dynamic Models With Stochastic Time-Dependent Parameters,” Water Resources Research, 45, 1–19.
Saltelli, A., Chan, K., and Scott, M. (eds.) (2000), Sensitivity Analysis, New York: Wiley.
Shmueli, G. (2010), “To Explain or to Predict?” Statistical Science, 25 (3), 289–310.
Smith, R., Tebaldi, C., Nychka, D., and Mearns, L. (2009), “Bayesian Modeling of Uncertainty in Ensembles of Climate Models,” Journal of the American Statistical Association, 104, 97–116.
Strong, M., Oakley, J. E., and Chilcott, J. (2011), “Managing Structural Uncertainty in Health Economic Decision Models: A Discrepancy Approach,” Journal of the Royal Statistical Society. Series C, Applied Statistics. doi:10.1111/j.1467-9876.2011.01014.x.
Vernon, I. R., Goldstein, M., and Bower, R. G. (2010), “Galaxy Formation: A Bayesian Uncertainty Analysis,” Bayesian Analysis, 5, 619–670.
Wei, G. C. G., and Tanner, M. A. (1990), “A Monte Carlo Implementation of the EM Algorithm and the Poor Man’s Data Augmentation Algorithms,” Journal of the American Statistical Association, 85 (411), 699–704.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wilkinson, R.D., Vrettas, M., Cornford, D. et al. Quantifying Simulator Discrepancy in Discrete-Time Dynamical Simulators. JABES 16, 554–570 (2011). https://doi.org/10.1007/s13253-011-0077-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13253-011-0077-3