Abstract
One of the well-known challenges in optimal experimental design is how to efficiently estimate the nested integrations of the expected information gain. The Gaussian approximation and associated importance sampling have been shown to be effective at reducing the numerical costs. However, they may fail due to the non-negligible biases and the numerical instabilities. A new approach is developed to compute the expected information gain, when the posterior distribution is multimodal—a situation previously ignored by the methods aiming at accelerating the nested numerical integrations. Specifically, the posterior distribution is approximated using a mixture distribution constructed by multiple runs of global search for the modes and weighted local Laplace approximations. Under any given probability of capturing all the modes, we provide an estimation of the number of runs of searches, which is dimension independent. It is shown that the novel global-local multimodal approach can be significantly more accurate and more efficient than the other existing approaches, especially when the number of modes is large. The methods can be applied to the designs of experiments with both calibrated and uncalibrated observation noises.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alexanderian A, Saibaba AK (2018) Efficient D-optimal design of experiments for infinite-dimensional Bayesian linear inverse problems. SIAM J Sci Comput 40(5):A2956–A2985
Alexanderian A, Petra N, Stadler G, Ghattas O (2014) A-optimal design of experiments for infinite-dimensional Bayesian linear inverse problems with regularized l0-sparsification. SIAM J Sci Comput 36(5):A2122–A2148
Atkinson AC, Donev AN (1992) Optimum experimental designs. Oxford University Press, Oxford
Atkinson A, Donev A, Tobias R et al (2007) Optimum experimental designs, with SAS, vol 34. Oxford University Press, Oxford
Beck J, Dia BM, Espath LF, Long Q, Tempone R (2018) Fast Bayesian experimental design: Laplace-based importance sampling for the expected information gain. Comput Methods Appl Mech Eng 334:523–553
Beck J, Dia BM, Espath LF, Tempone R (2020) Multilevel double loop monte carlo and stochastic collocation methods with importance sampling for Bayesian optimal experimental design. Int J Numer Meth Eng 121:3482
Bisetti F, Kim D, Knio O, Long Q, Tempone R (2016) Optimal Bayesian experimental design for priors of compact support with application to shock-tube experiments for combustion kinetics. Int J Numer Meth Eng 108:136–155
Busetto AG, Ong CS, Buhmann JM (2009) Optimized expected information gain for nonlinear dynamical systems. In: Proceedings of the 26th annual international conference on machine learning, ICML -09, p. 97-104. Association for Computing Machinery, New York, NY, USA
Calvetti D, Somersalo E (2007) Introduction to Bayesian scientific computing. Springer, Berlin
Carlon AG, Dia BM, Espath L, Lopez RH, Tempone R (2020) Nesterov-aided stochastic gradient methods using Laplace approximation for Bayesian design optimization. Comput Methods Appl Mech Eng 363:112909
Chaloner K, Verdinelli I (1995) Bayesian experimental design: a review. Stat Sci 10(3):273–304
Chen P, Villa U, Ghattas O (2019) Taylor approximation and variance reduction for pde-constrained optimal control under uncertainty. J Comput Phys 385:163–186
Dehideniya M, Overstall AM, Drovandi CC, McGree JM (2019) A synthetic likelihood-based Laplace approximation for efficient design of biological processes. arXiv:1903.04168
Dembo A, Zeitouni O (2010) Large deviations techniques and applications. Springer, Berlin
Ghosh S, Burnham KP, Laubscher NF, Dallal GE, Wilkinson L, Morrison DF, Loyer MW, Eisenberg B, Kullback S, Jolliffe IT, Simonoff JS (1987) Letters to the editor. Am Stat 41(4):338–341
Goda T, Hironaka T, Iwamoto T (2020) Multilevel monte carlo estimation of expected information gains. Stoch Anal Appl 38(4):581–600
Huan X, Marzouk YM (2013) Simulation-based optimal Bayesian experimental design for nonlinear systems. J Comput Phys 232(1):288–317
Ihler AT, Fisher JW, Moses RL, Willsky AS (2005) Nonparametric belief propagation for self-localization of sensor networks. IEEE J Sel Areas Commun 23(4):809–819
Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86
Lan S, Streets J, Shahbaba B (2014) Wormhole hamiltonian monte carlo. In: Proceedings of the Twenty-Eighth AAAI conference on artificial intelligence, AAAI’14, pp. 1953–1959. AAAI Press
Laplace PS (1986) Memoir on the probability of the causes of events. Stat Sci 1(3):364–378
Long Q, Scavino M, Tempone R, Wang S (2013) Fast estimation of expected information gains for Bayesian experimental designs based on Laplace approximations. Comput Methods Appl Mech Eng 259:24–39
Long Q, Motamed M, Tempone R (2015a) Fast Bayesian optimal experimental design for seismic source inversion. Comput Methods Appl Mech Eng 291:123–145
Long Q, Scavino M, Tempone R, Wang S (2015b) A Laplace method for under-determined Bayesian optimal experimental designs. Comput Methods Appl Mech Eng 285:849–876
Overstall AM, Woods DC (2017) Bayesian design of experiments using approximate coordinate exchange. Technometrics 59(4):458–470
Ryan KJ (2003) Estimating expected information gains for experimental designs with application to the random fatigue-limit model. J Comput Graph Stat 12(3):585–603
Ryan E, Drovandi C, Pettitt A (2015) Fully Bayesian experimental design for pharmacokinetic studies. Entropy 17:1063–1089
Schillings C, Sprungk B, Wacker P (2020) On the convergence of the Laplace approximation and noise-level-robustness of Laplace-based monte carlo methods for Bayesian inverse problems. Numer Math 145:915
Senarathne S, Drovandi C, McGree J (2020) A Laplace-based algorithm for Bayesian adaptive design. Stat Comput 30:1183–1208
Shaw JR, Bridges M, Hobson MP (2007) Efficient Bayesian inference for multimodal problems in cosmology. Mon Not R Astron Soc 378(4):1365–1370
Stein M (1987) Large sample properties of simulations using Latin hypercube sampling. Technometrics 29(2):143–151
Tak H, Meng XL, van Dyk DA (2018) A repelling-attracting Metropolis algorithm for multimodality. J Comput Graph Stat 27(3):479–490
Tarakanov A, Elsheikh AH (2020) Optimal bayesian experimental design for subsurface flow problems. Comput Methods Appl Mech Eng 370:113208
Tierney L, Kadane JB (1986) Accurate approximations for posterior moments and marginal densities. J Am Stat Assoc 81(393):82–86
Tierney L, Kass RE, Kadane JB (1989) Fully exponential Laplace approximations to expectations and variances of nonpositive functions. J Am Stat Assoc 84(407):710–716
Titterington DM (1975) Optimal design: some geometrical aspects of D-optimality. Biometrika 62(2):313–320
Verdinelli I (2000) A note on Bayesian design for the normal linear model with unknown error variance. Biometrika 87(1):222–227
Visser A, Slamet BA (2008) Including communication success in the estimation of information gain for multi-robot exploration. In: 2008 6th international symposium on modeling and optimization in mobile, Ad Hoc, and wireless networks and workshops, pp. 680–687
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Long, Q. Multimodal information gain in Bayesian design of experiments. Comput Stat 37, 865–885 (2022). https://doi.org/10.1007/s00180-021-01145-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-021-01145-9