Bayes Linear Emulation, History Matching, and Forecasting for Complex Computer Simulators

Abstract

Computer simulators are a useful tool for understanding complicated systems. However, any inferences made from them should recognize the inherent limitations and approximations in the simulator’s predictions for reality, the data used to run and calibrate the simulator, and the lack of knowledge about the best inputs to use for the simulator. This article describes the methods of emulation and history matching, where fast statistical approximations to the computer simulator (emulators) are constructed and used to reject implausible choices of input (history matching). Also described is a simple and tractable approach to estimating the discrepancy between simulator and reality induced by certain intrinsic limitations and uncertainties in the simulator and input data. Finally, a method for forecasting based on this approach is presented. The analysis is based on the Bayes linear approach to uncertainty quantification, which is similar in spirit to the standard Bayesian approach but takes expectation, rather than probability, as the primitive for the theory, with consequent simplifications in the prior uncertainty specification and analysis.

Appendix: Internal Discrepancy Perturbations

In this appendix, a description of the internal discrepancy experiments is provided. The first step is to identify potential quantities to perturb. For FUSE, the obvious quantities to choose are the two input time series and the initial condition. Also, the parameters could be perturbed at every time step. Similarly, the state vectors could be perturbed at every step, but this was not feasible in FUSE. Other possibilities not considered here include the time scale of the simulator and the accuracy of the numerical solver.

The next step is to informally assess the potential influence of each quantity. For example, increasing all rainfall by 10 % makes a large difference to the output, whereas increasing all evapotranspiration by 10 % makes a smaller but noticeable difference. Meanwhile, making large changes to the initial condition leads to extremely small changes away from the start of the simulation (recall that the quantities of interest are near the end of the simulation). From these initial explorations, each quantity can be categorized: if it has very little influence, it may not be worth perturbing; if it has a small influence, it may be worth including but not expending much effort on; if it has a large influence, it is worth carefully modeling. The outcome of this exploration for FUSE suggested that the initial condition was hardly relevant, the evapotranspiration was worth including, and the rainfall and parameter perturbations deserved more attention.

The final step is to consider how to generate perturbations of each quantity. The initial condition is ignored. For evapotranspiration, good estimates of observation uncertainty are lacking, but given the low influence of this quantity this is not too worrying: any plausible perturbation should be sufficient. Each evapotranspiration observation was multiplied by a perturbation drawn from a log-normal distribution, such that most observations were perturbed by no more than 10 %. Correlation between observations within 24 h was also included, so if a particular observation has a high perturbation, nearby ones will also. This is motivated by the daily period of the evapotranspiration time series.

Parameter perturbations are performed by multiplying each initial parameter by some random perturbation, with nearby multipliers being correlated. The size of the perturbations were chosen such that the parameters rarely changed by much more than 10 % over the course of a simulation. This creates collections of perturbations that cause the parameters to evolve slowly without sudden large changes and without a large change overall. The parameter perturbations have a significant effect on the output, but expert opinion about how these are likely to change over time and by how much is lacking. In principle, in such a situation one should make the correlation and the magnitude of the perturbations configurable parameters, so as to understand their influence. For this example, however, this complication is avoided. An example of the evolution of a particular choice of x ₍₁₎ for a particular perturbation can be seen in Fig. 2.6.

Fig. 2.6

The evolution of parameter x ₍₁₎ for a particular parameter perturbation

Perturbing the rainfall also has a significant effect on the output. In this case, however, there is some more guidance on the perturbations required. Sources of uncertainty in the rainfall was attributed to three significant causes: the “local” gauge measurement error, the process of aggregating readings to the nearest hour, and the process of averaging over the catchment by kriging. Suitable perturbations from these errors were generated and combined.

The overall rainfall perturbations generated for this process typically display occasional noticeable differences but mostly small differences. This suggests that the rainfall error could contribute significantly to discrepancy for maximum stream flow, but not so much for discrepancy for average stream flow.