Abstract
Deterministic computer models or simulators are used regularly to assist researchers in understanding the behavior of complex physical systems when real-world observations are limited. However, simulators are often imperfect representations of physical systems and may introduce layers of uncertainty into model-based inferences that are hard to quantify. To formalize the use of expert judgment in assessing simulator uncertainty, Goldstein and Rougier in J. Stat. Plan. Inference 139:1221–1239 (2009) propose a method, called reification, that decomposes the discrepancy between simulator predictions and reality by an improved, hypothetical computer model known as a “reified simulator”. One criticism of reification is that validation is, at best, challenging; only expert critiques can validate the subjective judgments used to specify a reified simulator. For this paper, we develop a procedure to quantify the advantages of reification for fast, modular simulators. The procedure is explained and implemented within the context of a rainfall-runoff that was developed by Iorgulescu, Beven, and Musy in Hydrol. Process. 19:2557–2573 (2005). We show that reification leads to informed judgments of simulator uncertainty
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bastos, L., and O’Hagan, A. (2009), “Diagnostics for Gaussian Process Emulators,” Technometrics, 51, 425–438.
Bhat, K., Haran, M., Tonkonojenkov, R., and Keller, K. (2009), “Inferring Likelihoods and Climate System Characteristics From Climate Models and Multiple Tracers,” Technical report, Pennsylvania State University, Department of Statistics.
Bratley, P., and Fox, B. (1988), “Algorithm 659: Implementing Sobol’s Quasirandom Sequence Generator,” ACM Transactions on Mathematical Software, 14 (1), 88–100.
Challenor, P. G. (2009), “Comment on ‘Reified Bayesian Modelling and Inference for Physical Systems’ (Pkg: P1221-1256),” Journal of Statistical Planning and Inference, 139 (3), 1248.
Craig, P., Goldstein, M., Rougier, J., and Seheult, A. (2001), “Bayesian Forecasting for Complex Systems Using Computer Simulators,” Journal of the American Statistical Association, 96 (454), 717–729.
Goldstein, M., and Rougier, J. (2004a), “Probabilistic Formulations for Transferring Inferences From Mathematical Models to Physical Systems,” SIAM Journal on Scientific Computing, 26 (2), 467–487.
— (2004b), “Probabilistic Formulations for Transferring Inferences From Mathematical Models to Physical Systems,” SIAM Journal on Scientific Computing, 26 (3), 467–487.
— (2009), “Reified Bayesian Modelling and Inference for Physical Systems (Pkg: P1221-1256),” Journal of Statistical Planning and Inference, 139 (3), 1221–1239.
Goldstein, M., and Woof, D. (2007), Bayes Linear Statistics, Chichester: Wiley.
Higdon, D., Gattiker, J., Williams, B., and Rightley, M. (2008), “Computer Model Calibration Using High Dimensional Output,” Journal of the American Statistical Association, 103 (482), 570–583.
Iorgulescu, I., Beven, K., and Musy, A. (2005), “Data-Based Modelling of Runoff and Chemical Tracer Concentrations in the Haute–Mentue Research Catchment (Switzerland),” Hydrological Processes, 19 (13), 2557–2573.
— (2007), “Flow, Mixing, and Displacement in Using a Data-Based Hydrochemical Model to Predict Conservative Tracer Data,” Water Resources Research, 43, 1–12.
Jeffreys, H. (1946), “An Invariant Form for the Prior Probability in Estimation Problem,” Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, 186 (1007), 453–461.
Kennedy, M., and O’Hagan, A. (2001), “Bayesian Calibration of Computer Models,” Journal of the Royal Statistical Society. Series B, Statistical Methodology, 63 (3), 425–464.
— (2000), “Predicting the Output From a Complex Computer Code When Fast Approximations Are Available,” Biometrika, 87 (1), 1–13.
Lavine, M., Hegerl, G., and Lozier, S. (2009), “Comment on ‘Reified Bayesian Modelling and Inference for Physical Systems’ (Pkg: P1221-1256),” Journal of Statistical Planning and Inference, 139 (3), 1243–1245.
Liu, F., Bayarri, M., and Berger, J. (2009), “Modularization in Bayesian Analysis, With Emphasis on Analysis of Computer Models,” Bayesian Analysis, 4 (1), 119–150.
Loeppky, J., Bingham, D., and Welch, W. (2006), “Computer Model Calibration or Tuning in Practice,” Technical Report 221, Department of Statistics, University of British Columbia,Vancouver, BC, V6T 1Z2.
Madsen, H. (2000), “Automatic Calibration of a Conceptual Rainfall-Runoff Model Using Multiple Objectives,” Journal of Hydrology, 235, 276–288.
Morita, S., Thall, P., and Mueller, P. (2007), “Determining the Effective Sample Size of a Parametric Prior,” Technical Report 36, UT MD Anderson Cancer Center Department of Biostatistics Working Paper Series.
Morris, M., and Higdon, D. (2009), “Comment on ‘Reified Bayesian Modelling and Inference for Physical Systems’ (Pkg: P1221-1256),” Journal of Statistical Planning and Inference, 139 (3), 1249–1250.
O’Hagan, A. (1978), “Curve Fitting and Optimal Design for Prediction,” Journal of the Royal Statistical Society. Series B. Methodological, 40, 1–24.
— (2006), “Reliability Engineering and System Safety,” The Fourth International Conference on Sensitivity Analysis of Model Output (SAMO 2004), 91 (10–11), 1290–1300.
Pina, J., Beven, K., and Freer, J. (1997), “Modelling the Hydrological Response of Mediterranean Catchments, Prades, Catalonia. The Use of Distributed Models as Aids to Hypothesis Formulation,” Hydrological Processes, 11, 1287–1306.
Rougier, J. (2008a), “Comment on Article by Sanso et al. Bayesian Analysis,” Bayesian Analysis, 3 (1), 45–56.
— (2008b), “Efficient Emulators for Multivariate Deterministic Functions,” Journal of Computational and Graphical Statistics, 17 (4), 827–843.
Sacks, J., Welch, W., Mitchell, T., and Wynn, H. (1989), “Design and Analysis of Computer Experiments (C/R: P423-435),” Statistical Science, 4, 409–423.
Sansó, B., Forest, C., and Zantedeschi, D. (2008), “Inferring Climate System Properties Using a Computer Model (Pkg: P1-62),” Bayesian Analysis, 3 (1), 1–38.
Santner, T., Williams, B., and Notz, W. (2003), The Design and Analysis of Computer Experiments, New York: Springer.
Smith, J. Q. (2009), “Comment on ‘Reified Bayesian Modelling and Inference for Physical Systems’ (Pkg: P1221-1256),” Journal of Statistical Planning and Inference, 139 (3), 1246–1247.
Sobol, L. (1967), “On the Distribution of Points in a Cube and the Approximate Evaluation of on the Distribution of Points in a Cube and the Approximate Evaluation of Integrals”, U.S.S.R. Computational Mathematics and Mathematical Physics 7, 86–112.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
House, L. Verifying Reification With Application to a Rainfall–Runoff Computer Simulator. JABES 16, 513–530 (2011). https://doi.org/10.1007/s13253-011-0075-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13253-011-0075-5