Quantifying Simulator Discrepancy in Discrete-Time Dynamical Simulators

  • Richard D. Wilkinson
  • Michail Vrettas
  • Dan Cornford
  • Jeremy E. Oakley


When making predictions with complex simulators it can be important to quantify the various sources of uncertainty. Errors in the structural specification of the simulator, for example due to missing processes or incorrect mathematical specification, can be a major source of uncertainty, but are often ignored. We introduce a methodology for inferring the discrepancy between the simulator and the system in discrete-time dynamical simulators. We assume a structural form for the discrepancy function, and show how to infer the maximum-likelihood parameter estimates using a particle filter embedded within a Monte Carlo expectation maximization (MCEM) algorithm. We illustrate the method on a conceptual rainfall-runoff simulator (logSPM) used to model the Abercrombie catchment in Australia. We assess the simulator and discrepancy model on the basis of their predictive performance using proper scoring rules. This article has supplementary material online.

Key Words

Model error Rainfall-runoff model Monte Carlo EM algorithm 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Beven, K. (2006), “A Manifesto for the Equifinality Thesis,” Journal of Hydrology, 320, 18–36. CrossRefGoogle Scholar
  2. Box, G. E. P., and Cox, D. R. (1964), “An Analysis of Transformations,” Journal of the Royal Statistical Society. Series B. Methodological, 26, 211–252. MATHMathSciNetGoogle Scholar
  3. Caffo, B. S., Jank, W., and Jones, G. L. (2005), “Ascent-Based Monte Carlo Expectation-Maximization,” Journal of the Royal Statistical Society. Series B. Methodological, 67 (2), 235–251. CrossRefMATHMathSciNetGoogle Scholar
  4. Conti, S., Gosling, J., Oakley, J., and O’Hagan, A. (2009), “Gaussian Process Emulation of Dynamic Computer Codes,” Biometrika, 96 (34), 663–676. CrossRefMATHMathSciNetGoogle Scholar
  5. Crucifix, M., and Rougier, J. (2009), “On the Use of Simple Dynamical Systems for Climate Predictions,” European Physical Journal, Special Topics, 174, 11–31. CrossRefGoogle Scholar
  6. Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977), “Maximum Likelihood from Incomplete Data Via the EM Algorithm,” Journal of the Royal Statistical Society. Series B. Methodological, 39, 1–38. MATHMathSciNetGoogle Scholar
  7. Doucet, A., de Freitas, N., and Gordon, N. (2001), Sequential Monte Carlo Methods in Practice, Berlin: Springer. MATHGoogle Scholar
  8. Gneiting, T., and Raftery, A. E. (2007), “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102 (477), 359–378. CrossRefMATHMathSciNetGoogle Scholar
  9. Godsill, S., Doucet, A., and West, M. (2004), “Monte Carlo Smoothing for Nonlinear Time Series,” Journal of the American Statistical Association, 99 (465), 156–168. CrossRefMATHMathSciNetGoogle Scholar
  10. Goldstein, M., and Rougier, J. (2009), “Reified Bayesian Modelling and Inference for Physical Systems” (with discussion), Journal of Statistical Planning and Inference, 139, 1221–1239. CrossRefMATHMathSciNetGoogle Scholar
  11. Gordon, N. J., Salmond, D. J., and Smith, A. F. M. (1993), “Novel Approach to Nonlinear/Non-Gaussian Bayesian State Estimation,” IEE Proceedings. Part F. Radar and Signal Processing, 140, 107–113. CrossRefGoogle Scholar
  12. Griffith, A. K., and Nichols, N. K. (2000), “Adjoint Techniques in Data Assimilation for Estimating Model Error,” Flow, Turbulence and Combustion, 65, 469–488. CrossRefMATHGoogle Scholar
  13. Higdon, D., Gattiker, J., Williams, B., and Rightley, M. (2008), “Computer Model Calibration Using High-Dimensional Output,” Journal of the American Statistical Association, 103, 570–583. CrossRefMATHMathSciNetGoogle Scholar
  14. House, L., Goldstein, M., and Rougier, J. (2008), “Assessing Model Discrepancy Using a Multi-model Ensemble,” MUCM Technical Report 08/07, Durham University and University of Bristol. Google Scholar
  15. Jolliffe, I. T., and Stephenson, D. B. (2003), Forecast Verification: A Practitioner’s Guide in Atmospheric Science, Chichester: Wiley and Sons. Google Scholar
  16. Kavetski, D., Kuczera, G., and Franks, S. W. (2003), “Semi-distributed Hydrological Modelling: a ‘Saturation Path’ Perspective on TOPMODEL and VIC,” Water Resources Research, 39, 1246–1253. CrossRefGoogle Scholar
  17. Kennedy, M. C., and O’Hagan, A. (2001), “A Bayesian Calibration of Computer Models” (with discussion), Journal of the Royal Statistical Society. Series B. Methodological, 63, 425–464. CrossRefMATHMathSciNetGoogle Scholar
  18. Kuczera, G., Kavetski, D., Franks, S., and Thyer, M. (2006), “Towards a Bayesian Total Error Analysis of Conceptual Rainfall-Runoff Models: Characterising Model Error Using Storm-Dependent Parameters,” Journal of Hydrology, 331, 161–177. CrossRefGoogle Scholar
  19. Meng, X. L., and Rubin, D. B. (1991), “Using EM to Obtain Asymptotic Variance-Covariance Matrices: The SEM Algorithm,” Journal of the American Statistical Association, 86, 899–909. CrossRefGoogle Scholar
  20. Nash, J. E., and Sutcliffe, J. V. (1970), “River Flow Forecasting Through Conceptual Models Part I—A Discussion of Principles,” Journal of Hydrology, 10, 282–290. CrossRefGoogle Scholar
  21. Oakley, J. E., and O’Hagan, A. (2002), “Bayesian Inference for the Uncertainty Distribution of Computer Model Outputs,” Biometrika, 89, 769–784. CrossRefGoogle Scholar
  22. Oberkampf, W. L., and Trucano, T. G. (2008), “Verification and Validation Benchmarks,” Nuclear Engineering and Design, 238, 716–743. Google Scholar
  23. Reichert, P., and Mieleitner, J. (2009), “Analyzing Input and Structural Uncertainty of Nonlinear Dynamic Models With Stochastic Time-Dependent Parameters,” Water Resources Research, 45, 1–19. CrossRefGoogle Scholar
  24. Saltelli, A., Chan, K., and Scott, M. (eds.) (2000), Sensitivity Analysis, New York: Wiley. MATHGoogle Scholar
  25. Shmueli, G. (2010), “To Explain or to Predict?” Statistical Science, 25 (3), 289–310. CrossRefMathSciNetGoogle Scholar
  26. Smith, R., Tebaldi, C., Nychka, D., and Mearns, L. (2009), “Bayesian Modeling of Uncertainty in Ensembles of Climate Models,” Journal of the American Statistical Association, 104, 97–116. CrossRefMathSciNetGoogle Scholar
  27. Strong, M., Oakley, J. E., and Chilcott, J. (2011), “Managing Structural Uncertainty in Health Economic Decision Models: A Discrepancy Approach,” Journal of the Royal Statistical Society. Series C, Applied Statistics. doi: 10.1111/j.1467-9876.2011.01014.x.
  28. Vernon, I. R., Goldstein, M., and Bower, R. G. (2010), “Galaxy Formation: A Bayesian Uncertainty Analysis,” Bayesian Analysis, 5, 619–670. CrossRefMathSciNetGoogle Scholar
  29. Wei, G. C. G., and Tanner, M. A. (1990), “A Monte Carlo Implementation of the EM Algorithm and the Poor Man’s Data Augmentation Algorithms,” Journal of the American Statistical Association, 85 (411), 699–704. CrossRefGoogle Scholar

Copyright information

© International Biometric Society 2011

Authors and Affiliations

  • Richard D. Wilkinson
    • 1
  • Michail Vrettas
    • 1
  • Dan Cornford
    • 2
  • Jeremy E. Oakley
    • 3
  1. 1.School of Mathematical SciencesUniversity of NottinghamNottinghamUK
  2. 2.Department of Engineering and Applied ScienceAston UniversityBirminghamUK
  3. 3.School of Mathematics and StatisticsUniversity of SheffieldSheffieldUK

Personalised recommendations