Abstract
Bayesian model selection objectively ranks competing models by computing Bayesian Model Evidence (BME) against test data. BME is the likelihood of data to occur under each model, averaged over uncertain parameters. Computing BME can be problematic: exact analytical solutions require strong assumptions; mathematical approximations (information criteria) are often strongly biased; assumption-free numerical methods (like Monte Carlo) are computationally impossible if the data set is large, for example like high-resolution snapshots from experimental movies. To use BME as ranking criterion in such cases, we develop the “Method of Forced Probabilities (MFP)”. MFP swaps the direction of evaluation: instead of comparing thousands of model runs on random model realizations with the observed movie snapshots, we force models to reproduce the data in each time step and record the individual probabilities of the model following these exact transitions. MFP is fast and accurate for models that fulfil the Markov property in time, paired with high-quality data sets that resolve all individual events. We demonstrate our approach on stochastic macro-invasion percolation models that simulate gas migration in porous media, and list additional examples of probable applications. The corresponding experimental movie was obtained from slow gas injection into water-saturated, homogeneous sand in a 25 x 25 x 1 cm acrylic glass tank. Despite the movie not always satisfying the high demands (resolving all individual events), we can apply MFP by suggesting a few workarounds. Results confirm that the proposed method can compute BME in previously unfeasible scenarios, facilitating a ranking among competing model versions for future model improvement.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Data Availability
The experimental data set used in this study is made available by [61] at Scholars Portal Dataverse: https://doi.org/10.5683/SP2/RQKOCN.
Code Availability
The modelling data and codes used in this study are available as a data set [62] in the DaRUS dataverse for Stochastic Simulation and Safety Research for Hydrosystems (LS3):https://doi.org/10.18419/darus-2815.
References
Schöniger, A, Wöhling, T, Samaniego, L, Nowak, W.: Model selection on solid ground: rigorous comparison of nine ways to evaluate Bayesian model evidence. Water Resour. Res. 50, 9484–9513 (2014). https://doi.org/10.1002/2014WR016062
Kashyap, RL: Optimal choice of AR and MA parts in autoregressive moving average models. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-4, 99–104 (1982). https://doi.org/10.1109/TPAMI.1982.4767213https://doi.org/10.1109/TPAMI.1982.4767213
Gideon, S.: Estimating the dimension of a model. Ann. Stat. 6(2), 461–464 (1978)
Hirotugu, A.: Information theory and an extension of the maximum likelihood principle. Paper presented at the Second International Symposium on Information Theory, Czaki. Akademiai Kiado, Budapest (1973)
Poeter, E, Anderson, D.: Multimodel ranking and inference in ground water modeling. Ground Water. 43, 597–605 (2005). https://doi.org/10.1111/j.1745-6584.2005.0061.x
Ye, M, Meyer, PD, Neuman, S.P.: On model selection criteria in multimodel analysis. Water Resour. Res. 44. https://doi.org/10.1029/2008WR006803 (2008)
Ye, M, Pohlmann, KF, Chapman, JB, Pohll, GM, Reeves, D.M.: A model-averaging method for assessing groundwater conceptual model uncertainty. Ground Water. 48, 716–728 (2010). https://doi.org/10.1111/j.1745-6584.2009.00633.x
Ye, M, Lu, D, Neuman, SP, Meyer, P.D.: Comment on “Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window” by Frank T.-C. Tsai and Xiaobao Li. Water Resour. Res., 46. https://doi.org/10.1029/2009WR008501https://doi.org/10.1029/2009WR008501 (2010)
Tsai, FTC, Li, X.: Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window. Water Resour. Res., 44. https://doi.org/10.1029/2007WR006576https://doi.org/10.1029/2007WR006576 (2008)
Tsai, FTC, Li, X.: Reply to comment by Ming Ye others. on “Inverse groundwater modeling for hydraulic conductivity estimation using Bayesian model averaging and variance window”. Water Resour. Res., 46. https://doi.org/10.1029/2009WR008591 (2010)
Singh, A, Mishra, S, Ruskauff, G.: Model averaging techniques for quantifying conceptual model uncertainty. Ground Water. 48, 701–715 (2010). https://doi.org/10.1111/j.1745-6584.2009.00642.xhttps://doi.org/10.1111/j.1745-6584.2009.00642.x
Morales-Casique, E, Neuman, SP, Vesselinov, V.V.: Maximum likelihood Bayesian averaging of airflow models in unsaturated fractured tuff using Occam and variance windows. Stoch. Environ. Res. Risk Assess. 24, 863–880 (2010). https://doi.org/10.1007/s00477-010-0383-2https://doi.org/10.1007/s00477-010-0383-2
Foglia, L, Mehl, SW, Hill, MC, Burlando, P.: Evaluating model structure adequacy: the case of the Maggia Valley groundwater system, southern Switzerland. Water Resour. Res. 49, 260–282 (2013). https://doi.org/10.1029/2011WR011779
Kloek, T, van Dijk, HK: Bayesian estimates of equation system parameters. An application of integration by Monte Carlo. Econometrica 46, 1–19 (1978). https://doi.org/10.2307/1913641
Zellner, A, Rossi, P.E.: Bayesian analysis of dichotomous quantal response models. J. Econom. 25, 365–393 (1984). https://doi.org/10.1016/0304-4076(84)90007-1https://doi.org/10.1016/0304-4076(84)90007-1
Snyder, C, Bengtsson, T, Bickel, P, Anderson, J.: Obstacles to high-dimensional particle filtering. Mon. Weather Rev. 136, 4629–4640 (2008). https://doi.org/10.1175/2008MWR2529.1
Bengtsson, T, Bickel, P, Li, B.: Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems. In: Nolan, D, Speed, T (eds.) Probability and Statistics: essays in Honor of David A. Freedman, vol. 2, pp 316–334. Institute of Mathematical Statistics, Beachwood (2008)
Zhang, Y, Liu, Y, Pau, G, Oladyshkin, S, Finsterle, S.: Evaluation of multiple reduced-order models to enhance confidence in global sensitivity analyses. Int. J. Greenhouse Gas Control. 49, 217–226 (2016). https://doi.org/10.1016/j.ijggc.2016.03.003
Xiu, D, Karniadakis, G.E.: The Wiener–Askey Polynomial Chaos for Stochastic Differential Equations. SIAM J. Sci. Comput. 24(2), 619–644 (2002). https://doi.org/10.1137/S1064827501387826
Willcox, K, Peraire, J.: Balanced model reduction via the proper orthogonal decomposition. AIAA J. 40(11), 2323–2330 (2002). https://doi.org/10.2514/2.1570
Kumar, R, Tolson, BA, Burn, DH: Review of surrogate modeling in water resources. Water Resour. Res. 48(7). https://doi.org/10.1029/2011WR011527 (2012)
Beaumont MA: Approximate Bayesian Computation in Evolution and Ecology. Annual Review of Ecology. Evolution, and Systematics. 41 (1), 379–406 (2010). https://doi.org/10.1146/annurev-ecolsys-102209-144621https://doi.org/10.1146/annurev-ecolsys-102209-144621
Mumford, KG, Hegele, PR, Vandenberg, GP: Comparison of two-dimensional and three-dimensional macroscopic invasion percolation simulations with laboratory experiments of gas bubble flow in homogeneous sands. Vadose Zone J. 14, 1–13 (2015). https://doi.org/10.2136/vzj2015.02.0028
Banerjee, I, Guthke, A, Van De Ven, CJC, Mumford, KG, Nowak, W: Overcoming the model-data-fit problem in porous media: a quantitative method to compare invasion-percolation models to high-resolution data. Water Resour. Res. 57, e2021WR029986 (2021). https://doi.org/10.1029/2021WR029986
Kass, RE, Raftery, AE.: Bayes factors. J. Amer. Stat. Assoc. 90, 773–795 (1995). https://doi.org/10.1080/01621459.1995.10476572
Hammersley, J.M.: Monte Carlo Methods for solving multivariable problems. Ann. New York Acad. Sci. 86, 844–874 (1960). https://doi.org/10.1111/j.1749-6632.1960.tb42846.x
Jeffreys, H.: Theory of Probability, 3rd edn. Clarendon Press, Oxford (1961)
Kolmogorov, A.: Foundations of the Theory of Probability. Morrison N, editor. Chelsea Publishing Company, New York (1950)
Gustafsson, F.: Particle filter theory and practice with positioning applications. IEEE Aerosp. Electron. Syst. Mag. 25(7), 53–82 (2010). https://doi.org/10.1109/MAES.2010.5546308
Ramgraber, M, Albert, C, Schirmer, M.: Data assimilation and online parameter optimization in groundwater modeling using nested particle filters. Water Resour. Res. 55(11), 9724–9747 (2019). https://doi.org/10.1029/2018WR024408
Nowak, W: Best unbiased ensemble linearization and the quasi-linear Kalman ensemble generator. Water Resour. Res. 45(4). https://doi.org/10.1029/2008WR007328 (2009)
Schöniger, A, Nowak, W, Hendricks Franssen, HJ: Parameter estimation by ensemble Kalman filters with transformed data: approach and application to hydraulic tomography. Water Resour. Res. 48(4). https://doi.org/10.1029/2011WR010462 (2012)
Ewing, RP, Berkowitz, B.: A generalized growth model for simulating initial migration of dense non-aqueous phase liquids. Water Resour. Res. 34, 611–622 (1998). https://doi.org/10.1029/97WR03754https://doi.org/10.1029/97WR03754
Van De Ven, CJC, Mumford, K.G.: Characterization of gas injection flow patterns subject to gravity and viscous forces. Vadose Zone J. 18, 1–11 (2019). https://doi.org/10.2136/vzj2019.02.0014
Broadbent, SR, Hammersley, J.M.: Percolation processes. Mathematical Proceedings of the Cambridge Philosophical Society. https://doi.org/10.1017/s0305004100032680 (1957)
Wilkinson, D, Willemsen JF.: Invasion percolation: a new form of percolation theory. J. Phys. A: Math. Gen. 16, 3365–3376 (1983). https://doi.org/10.1088/0305-4470/16/14/028
Wilkinson, D.: Percolation model of immiscible displacement in the presence of buoyancy forces. Phys. Rev. A. 30, 520–531 (1984). https://doi.org/10.1103/PhysRevA.30.520
Birovljev, A, Furuberg, L, Feder, J, Jssang, T, Mly, KJ, Aharony, A.: Gravity invasion percolation in two dimensions: experiment and simulation. Phys. Rev. Lett. 67, 584–587 (1991). https://doi.org/10.1103/PhysRevLett.67.584
Birovljev, A, Wagner, G, Meakin, P, Feder, J, Jøssang, T: Migration and fragmentation of invasion percolation clusters in two-dimensional porous media. Phys. Rev. E. 51, 5911–5915 (1995). https://doi.org/10.1103/PhysRevE.51.5911
Kueper, BH, McWhorter, D.B.: The use of macroscopic percolation theory to construct large-scale capillary pressure curves. Water Resour. Res. 28, 2425–2436 (1992). https://doi.org/10.1029/92WR01176https://doi.org/10.1029/92WR01176
Wagner, G, Meakin, P, Feder, J, Jøssang, T: Buoyancy-driven invasion percolation with migration and fragmentation. Physica A: Stat. Mech. Applic. 245, 217–230 (1997). https://doi.org/10.1016/S0378-4371(97)00324-5https://doi.org/10.1016/S0378-4371(97)00324-5
Ewing, RP, Berkowitz, B.: Stochastic pore-scale growth models of DNAPL migration in porous media. Adv. Water Resour. 24, 309–323 (2001). https://doi.org/10.1016/S0309-1708(00)00059-2
Glass, RJ, Conrad, SH, Yarrington, L.: Gravity-destabilized nonwetting phase invasion in macroheterogeneous porous media: near-pore-scale macro modified invasion percolation simulation of experiments. Water Resour. Res. 37, 1197–1207 (2001). https://doi.org/10.1029/2000WR900294https://doi.org/10.1029/2000WR900294
Mumford, KG, Smith, JE, Dickson, S.E.: The effect of spontaneous gas expansion and mobilization on the aqueous-phase concentrations above a dense non-aqueous phase liquid pool. Adv. Water Resour. 33, 504–513 (2010). https://doi.org/10.1016/j.advwatres.2010.02.002https://doi.org/10.1016/j.advwatres.2010.02.002
Trevisan, L, Illangasekare, TH, Meckel, T.A.: Modelling plume behavior through a heterogeneous sand pack using a commercial invasion percolation model. Geomech. Geophys. Geo-Energy Geo-Resour. 3, 327–337 (2017). https://doi.org/10.1007/s40948-017-0055-5https://doi.org/10.1007/s40948-017-0055-5
Molnar, IL, Mumford, KG, Krol MM.: Electro-thermal subsurface gas generation and transport: model validation and implications. Water Resour. Res. 55, 4630–4647 (2019). https://doi.org/10.1029/2018WR024095https://doi.org/10.1029/2018WR024095
Brooks, R, Corey, A.: Hydraulic properties of porous media. Water Resour. Res., 4 (1964)
Zhao, W, Ioannidis, MA.: Gas exsolution and flow during supersaturated water injection in porous media: I. Pore network modeling. Adv. Water Resour. 34, 2–14 (2011). https://doi.org/10.1016/j.advwatres.2010.09.010https://doi.org/10.1016/j.advwatres.2010.09.010
Gerhard, JI, Kueper, B.H.: Capillary pressure characteristics necessary for simulating DNAPL infiltration, redistribution, and immobilization in saturated porous media. Water Resources Research., 39. https://doi.org/10.1029/2002WR001270 (2003)
Ioannidis, MA, Chatzis, I, Dullien, F.A.L.: Macroscopic percolation model of immiscible displacement: effects of buoyancy and spatial structure. Water Resour. Res. 32(11), 3297–3310 (1996). https://doi.org/10.1029/95WR02216
Mumford, KG, Dickson, SE, Smith, J.E.: Slow gas expansion in saturated natural porous media by gas injection and partitioning with non-aqueous phase liquids. Adv. Water Resour. 32, 29–40 (2009). https://doi.org/10.1016/j.advwatres.2008.09.006
Schroth, MH, Istok, JD, Ahearn, SJ, Selker, J.S.: Characterization of Miller-similar silica sands for laboratory hydrologic studies. Soil Sci. Soc. Amer. J. 60, 1331–1339 (1996). https://doi.org/10.2136/sssaj1996.03615995006000050007x
Tidwell, VC, Glass, R.J.: X ray and visible light transmission for laboratory measurement of two-dimensional saturation fields in thin-slab systems. Water Resour. Res. 30(11), 2873–2882 (1994). https://doi.org/10.1029/94WR00953
Niemet, MR, Selker, J.S.: A new method for quantification of liquid saturation in 2D translucent porous media systems using light transmission. Adv. Water Resour. 24 (6), 651–666 (2001). https://doi.org/10.1016/S0309-1708(00)00045-2
Kechavarzi, C, Soga, K, Wiart, P.: Multispectral image analysis method to determine dynamic fluid saturation distribution in two-dimensional three-fluid phase flow laboratory experiments. Journal of Contaminant Hydrology. https://doi.org/10.1016/S0169-7722(00)00133-9https://doi.org/10.1016/S0169-7722(00)00133-9 (2000)
Gao, H, Tatomir, AB, Karadimitriou, NK, Steeb, H, Sauter, M.: Effects of surface roughness on the kinetic interface-sensitive tracer transport during drainage processes. Adv. Water Resour. 104044, 157 (2021). https://doi.org/10.1016/j.advwatres.2021.104044https://doi.org/10.1016/j.advwatres.2021.104044
Gonzalez-Nicolas, A, Schwientek, M, Sinsbeck, M, Nowak, W.: Characterization of export regimes in concentration–discharge plots via an advanced time-series model and event-based sampling strategies. Water, 13. https://doi.org/10.3390/w13131723 (2021)
Adrian, R.J.: Twenty years of particle image velocimetry. Exper. Fluids. 39, 159–169 (2005). https://doi.org/10.1007/s00348-005-0991-7https://doi.org/10.1007/s00348-005-0991-7
Gong, W, Gupta, HV, Yang, D, Sricharan, K, Hero, AO. III: Estimating epistemic and aleatory uncertainties during hydrologic modeling: an information theoretic approach. Water Resour. Res. 49 (4), 2253–2273 (2013). https://doi.org/10.1002/wrcr.20161
Shapiro, S.A.: Microseismicity - a Tool for Reservoir Characterization. EAGE Publications bv, Amsterdam (2008)
Van De Ven, C, Mumford, K, Banerjee, I: Replication Data for: Overcoming the model-data-fit problem in porous media: a quantitative method to compare invasion-percolation models to high-resolution data. Borealis. Available from: https://doi.org/10.5683/SP2/RQKOCNhttps://doi.org/10.5683/SP2/RQKOCN
Banerjee, I, Walter, P: Replication data for: The method of forced probabilities: a computation trick for Bayesian model evidence. DaRUS. Available from: https://doi.org/10.18419/darus-2815
Acknowledgements
The authors would like to thank the German Research Foundation (DFG) for financial support of this project within the Research Training Group GRK1829 “Integrated Hydrosystem Modelling” and the Cluster of Excellence EXC 2075 “Data-integrated Simulation Science (SimTech)” at the University of Stuttgart under Germany’s Excellence Strategy - EXC 2075 - 39074001. The authors would also like to thank Assistant Professor Dr. Cole Van De Ven, Carleton University, Canada, for his assistance with the provisioning, handling and processing of the experimental data used in this study.
Funding
Open Access funding enabled and organized by Projekt DEAL. The authors would like to thank the German Research Foundation (DFG) for financial support of this project within the Research Training Group GRK1829 “Integrated Hydrosystem Modelling” and the Cluster of Excellence EXC 2075 “Data-integrated Simulation Science (SimTech)” at the University of Stuttgart under Germany’s Excellence Strategy - EXC 2075 - 39074001.
Author information
Authors and Affiliations
Contributions
‘Not applicable’
Corresponding author
Ethics declarations
Ethics approval and consent to participate
‘Not applicable’
Consent for Publication
‘Not applicable’
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: Monte Carlo simulations grow exponentially in t max
We will prove here that the required size of MC simulations grows exponentially in tmax. Based on the tree structure in the didactic example from Section ??, we can see that there are \(N=2^{t_{max}}\) equiprobable branches. Therefore, the probability of the correct branch (the one with non-zero likelihood) is exactly:
which apparently is the situation-specific definition of BME. When approximating BME via MC sampling with i = 1…n independent random realizations, then each realization i has a constant and independent probability of finding or not finding the correct branch. This situation is described exactly by the Binomial distribution. The Binomial distribution is a discrete probability distribution of the number of success events k out of n independent trials:
where p is the probability of success and 1 − p is the probability of not obtaining success. Here, we define n as number of MC trials, X is the (random under repetition of the entire MC simulation) number of times the MC finds the correct branch \(\left (\text {i.e., the one with } Likelihood = 1\right )\). For a given execution of MC with given n, we will find k times the correct branch and n − k times any branch with Likelihood = 0.
Also, in the context of our example, the probability parameter p of the Binomial distribution is equal to BME = Ptrue. From the MC results, one would estimate:
Just for reassurance, the asymptotic MC result for BME at \(n \to \infty \) converges to the exact solution:
But can we estimate the MC error in this approximation, e.g., expressed as the coefficient of variation (CV). For the Binomial distribution, we know that:
As the conversion from k to the estimate of p is simply a division by n, we apply the rules of linearized uncertainty quantification to see that:
Using this in the definition of the coefficient of variation:
For small values of p as in the given example with \(p=2^{-t_{max}}\), we can replace 1 − p ≈ 1, and hence:
Thus, the number of MC runs required for a desired accuracy (expressed as a desired value of the CV) is:
This shows that the number n of required MC samples increases, for a given precision requirement, exponentially in tmax. The base 2 of the exponent originates from the tree structure, where each node expands into two further branches. In real applications, where the evolution of the model over time has more than two possibilities, the base will simply increase, so the exponential growth will be even stronger.
Appendix B: Tackling numerical instabilities in computation of BME
Here, we provide details on the approach of handling numerical instabilities in the BME computation when using MC integration (Eq. ??) for the uncertain parameters in Eq. ??.
To avoid very small likelihoods (BME values from the perspective of random events ω) turning into numerical zeros, we divide each sample likelihood by the maximum likelihood encountered in the whole ensemble, \(\max \limits \{ p(\mathbf {y}_{0} \mid \boldsymbol {\theta }_{k}, M_{k}) \}\), yielding values between 0 and 1. Then, Eq. ?? rewrites as:
Taking the logarithm and applying the MC approximation of the integral yields:
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Banerjee, I., Walter, P., Guthke, A. et al. The method of forced probabilities: a computation trick for Bayesian model evidence. Comput Geosci 27, 45–62 (2023). https://doi.org/10.1007/s10596-022-10179-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10596-022-10179-x