Skip to main content
SpringerLink
Log in

Surrogate-based branch-and-bound algorithms for simulation-based black-box optimization

Optimization and Engineering volume 24, pages 1463–1491 (2023)Cite this article

Abstract

Black-box surrogate-based optimization has received increasing attention due to the growing interest in solving optimization problems with embedded simulation data. The main challenge in surrogate-based optimization is the lack of consistently convergent behavior, due to the variability introduced by initialization, sampling, surrogate model selection, and training procedures. In this work, we build-up on our previously proposed data-driven branch-and-bound algorithm that is driven by adaptive sampling and the bounding of not entirely accurate surrogate models. This work incorporates Kriging and support vector regression surrogates, for which different bounding strategies are proposed. A variety of data-driven branching heuristics are also proposed and compared. The key finding of this work is that by bounding fitted, approximate surrogate models, one can employ a branch-and-bound structure that converges to the same optimum despite different initialization of samples and selection and training of a surrogate model. The performance of the algorithm is tested using box-constrained nonlinear benchmark problems with up to ten variables.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9.
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Abbreviations

\({f}^{*}\) :

Global solution to the problem

\({f}_{lb}^{*}\) :

The current best lower bound

\({f}_{ub}^{*}\) :

The current best upper bound

\({{f}_{lb}}_{k}\) :

The lower bound of node \(k\)

\({f}_{k}^{*}\) :

The upper bound of node \(k\)

\({x}_{lo}\) :

The lower bound of \(x\)

\({x}_{up}\) :

The upper bound of \(x\)

\(X\) :

Input samples

\(Y\) :

Output samples

\({X}_{k}\) :

Input samples of node \(k\)

\({Y}_{k}\) :

Output samples of node \(k\)

\({\varepsilon }_{a}\) :

Tolerance on the absolute gap

\({\varepsilon }_{r}\) :

Tolerance on the relative gap

\(p\) :

Perturbation on the surrogate model

References

  • Alarie S, Audet C, Gheribi AE, Kokkolaras M, Le Digabel S (2021) Two decades of blackbox optimization applications. EURO J Comput Optim 9:100011

    MathSciNet  MATH  Google Scholar 

  • Amaran S, Sahinidis NV, Sharda B, Bury SJ (2014) Simulation optimization: a review of algorithms and applications. 4OR 12:301–333

    MathSciNet  MATH  Google Scholar 

  • Androulakis IP, Maranas CD, Floudas CA (1995) αBB: a global optimization method for general constrained nonconvex problems. J Global Optim 7:337–363

    MathSciNet  MATH  Google Scholar 

  • Audet C, Côté-Massicotte J (2019) Dynamic improvements of static surrogates in direct search optimization. Optim Lett 13:1433–1447

    MathSciNet  MATH  Google Scholar 

  • Audet C, Jr JE, Dennis. (2006) Mesh adaptive direct search algorithms for constrained optimization. SIAM J Optim 17:188–217

    MathSciNet  MATH  Google Scholar 

  • Bajaj I, Faruque Hasan MM (2019) Deterministic global derivative-free optimization of black-box problems with bounded Hessian. Optim Lett 14:1011–1026

    MathSciNet  MATH  Google Scholar 

  • Bajaj I, Iyer SS, Faruque Hasan MM (2018) A trust region-based two phase algorithm for constrained black-box and grey-box optimization with infeasible initial point. Comput Chem Eng 116:306–321

    Google Scholar 

  • Bhosekar A, Ierapetritou M (2018) Advances in surrogate based modeling, feasibility analysis, and optimization: a review. Comput Chem Eng 108:250–267

    Google Scholar 

  • Boukouvala F, Floudas CA (2017) ARGONAUT: algorithms for global optimization of constrained grey-box computational problems. Optim Lett 11:895–913

    MathSciNet  MATH  Google Scholar 

  • Boukouvala F, Ierapetritou MG (2013) Surrogate-based optimization of expensive flowsheet modeling for continuous pharmaceutical manufacturing. J Pharm Innov 8:131–145

    Google Scholar 

  • Boukouvala F, Misener R, Floudas CA (2016) Global optimization advances in mixed-integer nonlinear programming, MINLP, and constrained derivative-free optimization, CDFO. Eur J Oper Res 252:701–727

    MathSciNet  MATH  Google Scholar 

  • Caballero JA, Grossmann IE (2008) An algorithm for the use of surrogate models in modular flowsheet optimization. AIChE J 54:2633–2650

    Google Scholar 

  • Cherkassky V, Ma Y (2004) Practical selection of SVM parameters and noise estimation for SVM regression. Neural Netw 17:113–126

    MATH  Google Scholar 

  • Clarke SM, Griebsch JH, Simpson TW (2004) Analysis of support vector regression for approximation of complex engineering analyses. J Mech Des 127:1077–1087

    Google Scholar 

  • Conn AR, Scheinberg K, Vicente LN (2009) Introduction to derivative-free optimization (SIAM)

  • Cox DD, John S (1992) A statistical method for global optimization. In [Proceedings] 1992 IEEE international conference on systems, man, and cybernetics, vol 2, pp 1241–46

  • Cozad A, Sahinidis NV, Miller DC (2014) Learning surrogate models for simulation-based optimization. AIChE J 60:2211–2227

    Google Scholar 

  • Eason J, Cremaschi S (2014) Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput Chem Eng 68:220–232

    Google Scholar 

  • García-García JC, García-Ródenas R, Codina E (2020) A surrogate-based cooperative optimization framework for computationally expensive black-box problems. Optim Eng 21:1053–1093

    MathSciNet  MATH  Google Scholar 

  • Garud SS, Mariappan N, Karimi IA (2019) Surrogate-based black-box optimisation via domain exploration and smart placement. Comput Chem Eng 130:106567

    Google Scholar 

  • Gilmore P, Kelley CT (1995) An implicit filtering algorithm for optimization of functions with many local minima. SIAM J Optim 5:269–285

    MathSciNet  MATH  Google Scholar 

  • Henao CA, Maravelias CT (2011) Surrogate-based superstructure optimization framework. AIChE J 57:1216–1232

    Google Scholar 

  • Hüllen G, Zhai J, Kim SH, Sinha A, Realff MJ, Boukouvala F (2019) Managing uncertainty in data-driven simulation-based optimization. Comput Chem Eng 136:106519

    Google Scholar 

  • Huyer W, Neumaier A (1999) Global optimization by multilevel coordinate search. J Global Optim 14:331–355

    MathSciNet  MATH  Google Scholar 

  • Huyer W, Neumaier A (2008) “SNOBFIT – stable noisy optimization by branch and fit”, ACM trans. Math Softw 35:1–25

    Google Scholar 

  • Johnson SG (2018) The NLopt nonlinear-optimization package. http://ab-initio.mit.edu/nlopt

  • Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Global Optim 21:345–383

    MathSciNet  MATH  Google Scholar 

  • Jones DR (2009) Direct global optimization algorithm. In: Floudas CA, Pardalos PM (eds) Encyclopedia of optimization. Springer, US, Boston, MA

    Google Scholar 

  • Jones DR, Perttunen CD, Stuckman BE (1993) Lipschitzian optimization without the Lipschitz constant. J Optim Theory Appl 79:157–181

    MathSciNet  MATH  Google Scholar 

  • Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Global Optim 13:455–492

    MathSciNet  MATH  Google Scholar 

  • Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of ICNN'95: international conference on neural networks, 1942–48 vol 4

  • Kieslich CA, Boukouvala F, Floudas CA (2018) Optimization of black-box problems using Smolyak grids and polynomial approximations. J Global Optim 71:845–869

    MathSciNet  MATH  Google Scholar 

  • Kim SH, Boukouvala F (2020) Machine learning-based surrogate modeling for data-driven optimization: a comparison of subset selection for regression techniques. Optim Lett 14:989–1010

    MathSciNet  MATH  Google Scholar 

  • Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. Science 220:671–680

    MathSciNet  MATH  Google Scholar 

  • Kolda TG, Lewis RM, Torczon V (2003) Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev 45:385–482

    MathSciNet  MATH  Google Scholar 

  • Kontogiannis SG, Savill MA (2020) A generalized methodology for multidisciplinary design optimization using surrogate modelling and multifidelity analysis. Optim Eng 21:723–759

    MathSciNet  Google Scholar 

  • Kushner HJ (1964) A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise

  • Larson J, Leyffer S, Palkar P, Wild SM (2021) A method for convex black-box integer global optimization. J Global Optim 80:439–477

    MathSciNet  MATH  Google Scholar 

  • Lewis RM, Torczon V (1999) Pattern search algorithms for bound constrained minimization. SIAM J Optim 9:1082–1099

    MathSciNet  MATH  Google Scholar 

  • Manno A, Amaldi E, Casella F, Martelli E (2020) A local search method for costly black-box problems and its application to CSP plant start-up optimization refinement. Optim Eng 21:1563–1598

    MathSciNet  MATH  Google Scholar 

  • McBride K, Sundmacher K (2019) Overview of surrogate modeling in chemical process engineering. Chem Ing Tec 91:228–239

    Google Scholar 

  • McKay MD, Beckman RJ, Conover WJ (1979) Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21:239–245

    MathSciNet  MATH  Google Scholar 

  • Meyer D, Leisch F, Hornik K (2003) The support vector machine under test. Neurocomputing 55:169–186

    Google Scholar 

  • Misener R, Floudas CA (2014) ANTIGONE: algorithms for continuous/integer global optimization of nonlinear equations. J Global Optim 59:503–526

    MathSciNet  MATH  Google Scholar 

  • Müller J, Park J, Sahu R, Varadharajan C, Arora B, Faybishenko B, Agarwal D (2021) Surrogate optimization of deep neural networks for groundwater predictions. J Global Optim 81:203–231

    MathSciNet  MATH  Google Scholar 

  • Nelder JA, Mead R (1965) A simplex method for function minimization. Comput J 7:308–313

    MathSciNet  MATH  Google Scholar 

  • Powell MJD (2002) UOBYQA: unconstrained optimization by quadratic approximation. Math Program 92:555–582

    MathSciNet  MATH  Google Scholar 

  • Puranik Y, Sahinidis NV (2017) Bounds tightening based on optimality conditions for nonconvex box-constrained optimization. J Global Optim 67:59–77

    MathSciNet  MATH  Google Scholar 

  • Reeves CR (1997) Feature article—genetic algorithms for the operations researcher. INFORMS J Comput 9:231–250

    MATH  Google Scholar 

  • Rios LM, Sahinidis NV (2013) Derivative-free optimization: a review of algorithms and comparison of software implementations. J Global Optim 56:1247–1293

    MathSciNet  MATH  Google Scholar 

  • Ryoo HS, Sahinidis NV (1996) A branch-and-reduce approach to global optimization. J Global Optim 8:107–138

    MathSciNet  MATH  Google Scholar 

  • Schweidtmann AM, Mitsos A (2019) Deterministic global optimization with artificial neural networks embedded. J Optim Theory Appl 180:925–948

    MathSciNet  MATH  Google Scholar 

  • Schweidtmann AM, Bongartz D, Grothe D, Kerkenhoff T, Lin X, Najman J, Mitsos A (2021) Deterministic global optimization with Gaussian processes embedded. Math Program Comput 13:553–581

    MathSciNet  MATH  Google Scholar 

  • Schweidtmann AM, Weber JM, Wende C, Netze L, Mitsos A (2022) Obey validity limits of data-driven models through topological data analysis and one-class classification. Optim Eng 23:855–876

    MathSciNet  MATH  Google Scholar 

  • Shahriari B, Swersky K, Wang Z, Adams RP, De Freitas N (2015) Taking the human out of the loop: a review of Bayesian optimization. Proc IEEE 104:148–175

    Google Scholar 

  • Shukla HA, de Avila Ferreira T, Faulwasser T, Bonvin D, Jones CN (2020) Convergence certificate for stochastic derivative-free trust-region methods based on Gaussian processes. arXiv preprint arXiv:2010.01120

  • Stein M (1987) Large sample properties of simulations using latin hypercube sampling. Technometrics 29:143–151

    MathSciNet  MATH  Google Scholar 

  • Tawarmalani M, Sahinidis NV (2004) Global optimization of mixed-integer nonlinear programs: a theoretical and computational study. Math Program 99:563–591

    MathSciNet  MATH  Google Scholar 

  • Tawarmalani M, Sahinidis NV (2005) A polyhedral branch-and-cut approach to global optimization. Math Program 103:225–249

    MathSciNet  MATH  Google Scholar 

  • Thebelt A, Kronqvist J, Mistry M, Lee RM, Sudermann-Merx N, Misener R (2021) ENTMOOT: a framework for optimization over ensemble tree models. Comput Chem Eng 151:107343

    Google Scholar 

  • Torczon V (1997) On the convergence of pattern search algorithms. SIAM J Optim 7:1–25

    MathSciNet  MATH  Google Scholar 

  • Ughi G, Abrol V, Tanner J (2021) An empirical study of derivative-free-optimization algorithms for targeted black-box attacks in deep neural networks. Optim Eng.

  • Vapnik V (1999) The nature of statistical learning theory. Springer, New York

    MATH  Google Scholar 

  • Vapnik V, Kotz S (2006) Estimation of dependences based on empirical data. Springer, Berlin

    Google Scholar 

  • Wächter A, Biegler LT (2006) On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math Program 106:25–57

    MathSciNet  MATH  Google Scholar 

  • Wang GG, Shan S (2006) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129:370–380

    Google Scholar 

  • Whitley D (1994) A genetic algorithm tutorial. Stat Comput 4:65–85

    Google Scholar 

  • Xia W, Shoemaker C (2021) GOPS: efficient RBF surrogate global optimization algorithm with high dimensions and many parallel processors including application to multimodal water quality PDE model calibration. Optim Eng 22:2741–2777

    MathSciNet  MATH  Google Scholar 

  • Xu WL, Nelson BL (2013) Empirical stochastic branch-and-bound for optimization via simulation. IIE Trans 45:685–698

    Google Scholar 

  • Zhai J, Boukouvala F (2019) Nonlinear variable selection algorithms for surrogate modeling. AIChE J 65:e16601

    Google Scholar 

  • Zhai J, Boukouvala F (2022) Data-driven spatial branch-and-bound algorithms for box-constrained simulation-based optimization. J Global Optim 82:21–50

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors acknowledge financial support from the National Science Foundation (NSF 1805724, NSF 1944678) (JZ, FB), RAPID AIChE/DOE Synopsis Project (FB) and Georgia Institute of Technology Startup Funding (JZ, FB).

Author information

Authors and Affiliations

  1. School of Chemical and Biomolecular Engineering, Georgia Institute of Technology, 311 Ferst Dr, Atlanta, GA, 30332, USA

    Jianyuan Zhai & Fani Boukouvala

Authors
  1. Jianyuan Zhai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fani Boukouvala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fani Boukouvala.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhai, J., Boukouvala, F. Surrogate-based branch-and-bound algorithms for simulation-based black-box optimization. Optim Eng 24, 1463–1491 (2023). https://doi.org/10.1007/s11081-022-09740-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11081-022-09740-5

Keywords

search

Navigation