Abstract
Bayesian optimization (BO) with Gaussian processes (GP) as surrogate models is widely used to optimize analytically unknown and expensive-to-evaluate functions. In this paper, we propose Prior-mean-RObust Bayesian Optimization (PROBO) that outperforms classical BO on specific problems. First, we study the effect of the Gaussian processes’ prior specifications on classical BO’s convergence. We find the prior’s mean parameters to have the highest influence on convergence among all prior components. In response to this result, we introduce PROBO as a generalization of BO that aims at rendering the method more robust towards prior mean parameter misspecification. This is achieved by explicitly accounting for GP imprecision via a prior near-ignorance model. At the heart of this is a novel acquisition function, the generalized lower confidence bound (GLCB). We test our approach against classical BO on a real-world problem from material science and observe PROBO to converge faster. Further experiments on multimodal and wiggly target functions confirm the superiority of our method.
Keywords
- Bayesian optimization
- Imprecise Gaussian process
- Imprecise probabilities
- Prior near-ignorance
- Model imprecision
- Robust optimization
Julian Rodemann would like to thank the scholarship program of Evangelisches Studienwerk Villigst for the support of his studies and Lars Kotthoff for providing data as well as Christoph Jansen and Georg Schollmeyer for valuable remarks.
Open Science: Code to reproduce all findings and figures presented in this paper is available on a public repository: github.com/rodemann/gp-imprecision-in-bo.
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Also referred to as infill criterion.
- 2.
Focus search shrinks the search space and applies random search, see [4, p. 7].
- 3.
BO’s computational complexity depends on the SM. In case of GPs, it is \(\mathcal {O}(n^{3})\) due to the required inversion of the covariance matrix, where n is total number of target function evaluations.
- 4.
Also called covariance function or kernel function.
- 5.
To the best of our knowledge, this is the very first systematic assessment of GP prior’s influence on BO.
- 6.
- 7.
Note that from a decision-theoretic point of view, LCB violates the dominance principle. GLCB inherits this property.
- 8.
Further note that with expensive target functions to optimize, the computational costs of surrogate models and acquisition functions in BO can be regarded as negligible. The computational complexity of PROBO is the same as for BO with GP.
References
Augustin, T., Coolen, F.P., de Cooman, G., Troffaes, M.C.M.: Introduction to Imprecise Probabilities. Wiley, Chichester (2014)
Awal, M.A., Masud, M., Hossain, M.S., Bulbul, A.A., Mahmud, S.M.H., Bairagi, A.K.: A novel Bayesian optimization-based machine learning framework for COVID-19 detection from inpatient facility data. IEEE Access 9, 10263–10281 (2021)
Benavoli, A., Zaffalon, M.: Prior near ignorance for inferences in the k-parameter exponential family. Statistics 49(5), 1104–1140 (2015)
Bischl, B., Richter, J., Bossek, J., Horn, D., Thomas, J., Lang, M.: mlrMBO: a modular framework for model-based optimization of expensive black-box functions. arXiv preprint arXiv:1703.03373 (2017)
Bossek, J.: smoof: Single- and multi-objective optimization test functions. R J. 9(1), 103–113 (2017)
Cox, D.D., John, S.: A statistical method for global optimization. In: Proceedings of 1992 IEEE International Conference on Systems, Man, and Cybernetics, pp. 1241–1246. IEEE (1992)
Duvenaud, D.: Automatic model construction with Gaussian processes. Ph.D. thesis. University of Cambridge (2014)
Frazier, P.I., Wang, J.: Bayesian optimization for materials design. In: Lookman, T., Alexander, F.J., Rajan, K. (eds.) Information Science for Materials Discovery and Design. SSMS, vol. 225, pp. 45–75. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-23871-5_3
Horn, D., Wagner, T., Biermann, D., Weihs, C., Bischl, B.: Model-based multi-objective optimization: taxonomy, multi-point proposal, toolbox and benchmark. In: Gaspar-Cunha, A., Henggeler Antunes, C., Coello, C.C. (eds.) EMO 2015. LNCS, vol. 9018, pp. 64–78. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15934-8_5
Malkomes, G., Garnett, R.: Automating Bayesian optimization with Bayesian optimization. Adv. Neural. Inf. Process. Syst. 31, 5984–5994 (2018)
Mangili, F.: A prior near-ignorance Gaussian process model for nonparametric regression. In: ISIPTA 2015: Proceedings of the 9th International Symposium on Imprecise Probability: Theories and Applications, pp. 187–196 (2015)
Mangili, F.: A prior near-ignorance Gaussian process model for nonparametric regression. Int. J. Approx. Reason. 78, 153–171 (2016)
McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 42(1), 55–61 (2000)
Močkus, J.: On Bayesian methods for seeking the extremum. In: Marchuk, G.I. (ed.) Optimization Techniques 1974. LNCS, vol. 27, pp. 400–404. Springer, Heidelberg (1975). https://doi.org/10.1007/3-540-07165-2_55
Nguyen, V.: Bayesian optimization for accelerating hyper-parameter tuning. In: 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pp. 302–305. IEEE (2019)
Pyzer-Knapp, E.O.: Bayesian optimization for accelerated drug discovery. IBM J. Res. Dev. 62(6), 2:1-2:7 (2018)
R Core Team: R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2020)
Rasmussen, C.E.: Gaussian processes in machine learning. In: Bousquet, O., von Luxburg, U., Rätsch, G. (eds.) ML -2003. LNCS (LNAI), vol. 3176, pp. 63–71. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28650-9_4
Insua, D.R., Ruggeri, F.: Robust Bayesian Analysis. Springer, New York (2000)
Rodemann, J.: Robust generalizations of stochastic derivative-free optimization. Master’s thesis. LMU Munich (2021)
Schmidt, A.M., Conceição, M.F.G., Moreira, G.A.: Investigating the sensitivity of Gaussian processes to the choice of their correlation function and prior specifications. J. Stat. Comput. Simul. 78(8), 681–699 (2008)
Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. Adv. Neural. Inf. Process. Syst. 25, 2951–2959 (2012)
Utkin, L., Kovalev, M., Meldo, A., Coolen, F.: Imprecise extensions of random forests and random survival forests. In: International Symposium on Imprecise Probabilities: Theories and Applications, pp. 404–413. PMLR (2019)
Wahab, H., Jain, V., Tyrrell, A.S., Seas, M.A., Kotthoff, L., Johnson, P.A.: Machine-learning-assisted fabrication: Bayesian optimization of laser-induced graphene patterning using in-situ Raman analysis. Carbon 167, 609–619 (2020)
Walter, G., Augustin, T.: Imprecision and prior-data conflict in generalized Bayesian inference. J. Stat. Theory Pract. 3(1), 255–271 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Rodemann, J., Augustin, T. (2022). Accounting for Gaussian Process Imprecision in Bayesian Optimization. In: Honda, K., Entani, T., Ubukata, S., Huynh, VN., Inuiguchi, M. (eds) Integrated Uncertainty in Knowledge Modelling and Decision Making. IUKM 2022. Lecture Notes in Computer Science(), vol 13199. Springer, Cham. https://doi.org/10.1007/978-3-030-98018-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-98018-4_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98017-7
Online ISBN: 978-3-030-98018-4
eBook Packages: Computer ScienceComputer Science (R0)