Skip to main content
SpringerLink
Log in

Branch-and-Model: a derivative-free global optimization algorithm

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

This paper presents a novel derivative-free global optimization algorithm Branch-and-Model (BAM). The BAM algorithm partitions the search domain dynamically, builds surrogate models around carefully selected evaluated points, and uses these models to exploit local function trends and speed up convergence. For model construction, BAM employs Automated Learning of Algebraic Models (ALAMO). The ALAMO algorithm generates algebraic models of the black-box function using various base functions and selection criteria. BAM’s potentially optimal identification scheme saves computational effort and prevents delays in searching for optimal solutions. The BAM algorithm is guaranteed to converge to the globally optimal function value under mild assumptions. Extensive computational experiments over 500 publicly open-source test problems and one industrially-relevant application show that BAM outperforms state-of-the-art DFO algorithms regardless of problem convexity and smoothness, especially for higher-dimensional problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The problems utilized in this study are all available from [5, 30].

References

  1. Audet, C., Dennis, J.E., Jr.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17, 188–217 (2006)

    MathSciNet  MATH  Google Scholar 

  2. Audet, C., Le Digabel, S., Tribes, C., Montplaisir, V.R.: The NOMAD project (current as of 19 September, 2022). Software available at http://www.gerad.ca/nomad

  3. Campana, E.F., Diez, M., Iemma, U., Liuzzi, G., Lucidi, S., Rinaldi, F., Serani, A.: Derivative-free global ship design optimization using global/local hybridization of the DIRECT algorithm. Optim. Eng. 17(1), 127–156 (2016)

    MathSciNet  MATH  Google Scholar 

  4. Cappellari, M., Verolme, E.K., Marel, R.P.V.D., Kleijn, G.V., Illingworth, G.D., Franx, M., Carollo, C.M., Zeeuw, P.T.D.: The counterrotating core and the black hole mass of IC 1459. Astrophys. J. 578(2), 787 (2002)

    Google Scholar 

  5. Comparison of derivative-free optimization algorithms. https://sahinidis.coe.gatech.edu/bbo?q=dfo

  6. Conn, A.R., Gould, N., Lescrenier, M., Toint, P.L.: Performance of a multifrontal scheme for partially separable optimization. In: Gomez, S., Hennart, J.-P. (eds.) Advances in Optimization and Numerical Analysis, pp. 79–96. Kluwer Academic Publishers, Dordrecht (1994)

    MATH  Google Scholar 

  7. Conn, A.R., Scheinberg, K., Toint, P.L.: On the convergence of derivative-free methods for unconstrained optimization. In: Buhmann, M.D., Iserles, A. (eds.) Approximation Theory and Optimization, Tribute to M. J. D. Powell, pp. 83–108. Cambridge University Press, Cambridge (1996)

    Google Scholar 

  8. Cozad, A., Sahinidis, N.V., Miller, D.C.: Learning surrogate models for simulation-based optimization. AIChE J. 60(6), 2211–2227 (2014)

    Google Scholar 

  9. Custódio, A., Scheinberg, K., Vicente, L.N.: Methodologies and software for derivative-free optimization. In: SIAM Journal on Advances and Trends in Optimization with Engineering Applications, pp. 495–506 (2017)

  10. Finkel, D., Kelley, C.T.: An adaptive restart implementation of DIRECT. Tech. rep., North Carolina State University. Center for Research in Scientific Computation (2004)

  11. Gablonsky, J.M., Kelley, C.T.: A locally-biased form of the DIRECT algorithm. J. Glob. Optim. 21, 27–37 (2001)

    MathSciNet  MATH  Google Scholar 

  12. Gaviano, M., Kvasov, D.E., Lera, D., Sergeyev, Y.D.: Algorithm 829: software for generation of classes of test functions with known local and global minima for global optimization. ACM Trans. Math. Softw. (TOMS) 29(4), 469–480 (2003)

    MathSciNet  MATH  Google Scholar 

  13. Gilmore, P., Kelley, C.T.: An implicit filtering algorithm for optimization of functions with many local minima. SIAM J. Optim. 5, 269–285 (1995)

    MathSciNet  MATH  Google Scholar 

  14. GLOBAL Library. http://www.gamsworld.org/global/globallib.htm

  15. Hansen, N.: The CMA evolution strategy: a tutorial. http://www.lri.fr/~hansen/cmaesintro.html

  16. Hare, W., Nutini, J., Tesfamariam, S.: A survey of non-gradient optimization methods in structural engineering. Adv. Eng. Softw. 59, 19–28 (2013)

    Google Scholar 

  17. Hayes, R.E., Bertrand, F.H., Audet, C., Kolaczkowski, S.T.: Catalytic combustion kinetics: using a direct search algorithm to evaluate kinetic parameters from light-off curves. Can. J. Chem. Eng. 81, 1192–1199 (2003)

    Google Scholar 

  18. Holmström, K., Quttineh, N.H., Edvall, M.M.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization. Optim. Eng. 9, 311–339 (2008)

    MathSciNet  MATH  Google Scholar 

  19. Huyer, W., Neumaier, A.: Global optimization by multilevel coordinate search. J. Glob. Optim. 14(4), 331–355 (1999)

    MathSciNet  MATH  Google Scholar 

  20. Huyer, W., Neumaier, A.: Snobfit-stable noisy optimization by branch and fit. ACM Trans. Math. Softw. (TOMS) 35(2), 1–25 (2008)

    MathSciNet  Google Scholar 

  21. Jones, D.J., Martins, J.R.: The DIRECT algorithm: 25 years Later. J. Glob. Optim. 79(3), 521–566 (2021)

    MathSciNet  MATH  Google Scholar 

  22. Jones, D.R., Perttunen, C.D., Stuckman, B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79, 157–181 (1993)

    MathSciNet  MATH  Google Scholar 

  23. Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13, 455–492 (1998)

    MathSciNet  MATH  Google Scholar 

  24. Khajavirad, A., Sahinidis, N.V.: A hybrid LP/NLP paradigm for global optimization relaxations. Math. Program. Comput. 10, 383–421 (2018)

    MathSciNet  MATH  Google Scholar 

  25. Lin, Y., Schrage, L.: The global solver in the LINDO API. Optim. Methods Softw. 24(4–5), 657–668 (2009)

    MathSciNet  MATH  Google Scholar 

  26. Liu, H., Xu, S., Wang, X., Wu, J., Song, Y.: A global optimization algorithm for simulation-based problems via the extended DIRECT scheme. Eng. Optim. 47(11), 1441–1458 (2015)

    MathSciNet  Google Scholar 

  27. Liu, Q., Zeng, J., Yang, G.: MrDIRECT: a multilevel robust DIRECT algorithm for global optimization problems. J. Glob. Optim. 62(2), 205–227 (2015)

    MathSciNet  MATH  Google Scholar 

  28. Ljungberg, K., Holmgren, S., Carlborg, Ö.: Simultaneous search for multiple QTL using the global optimization algorithm DIRECT. Bioinformatics 20(12), 1887–1895 (2004)

    Google Scholar 

  29. Lukšan, L., Vlček, J.: Test problems for nonsmooth unconstrained and linearly constrained optimization. Tech. rep., Institute of Computer Science, Academy of Sciences of the Czech Republic (2000). http://www3.cs.cas.cz/ics/reports/v798-00.ps

  30. Ma, K., Sahinidis, N.V., Bindlish, R., Bury, S.J., Haghpanah, R., Rajagopalan, S.: Data-driven strategies for extractive distillation unit optimization. Comput. Chem. Eng. 167, 107970 (2022)

    Google Scholar 

  31. Matheron, G.: Principles of geostatistics. Econ. Geol. 58, 1246–1266 (1967)

    Google Scholar 

  32. McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21, 239–245 (1979)

    MathSciNet  MATH  Google Scholar 

  33. Mockus, J.: On the pareto optimality in the context of Lipschitzian optimization. Informatica 22(4), 521–536 (2011)

    MathSciNet  MATH  Google Scholar 

  34. Mockus, J., Paulavičius, R., Rusakevičius, D., Šešok, D., Žilinskas, J.: Application of Reduced-set Pareto-Lipschitzian Optimization to truss optimization. J. Glob. Optim. 67(1–2), 425–450 (2017)

    MathSciNet  MATH  Google Scholar 

  35. Moré, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)

    MathSciNet  MATH  Google Scholar 

  36. Munack, H.: On global optimization using interval arithmetic. Computing 48(3), 319–336 (1992)

    MathSciNet  MATH  Google Scholar 

  37. Myers, R.H., Montgomery, D.C., Anderson-Cook, C.M.: Response Surface Methodology: Process and Product Optimization Using Designed Experiments. Wiley Series in Probability and Statistics. Wiley (2016)

    MATH  Google Scholar 

  38. Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)

    MathSciNet  MATH  Google Scholar 

  39. Nesterov, Y.: Gradient methods for minimizing composite objective function. Math. Program. Ser. B 140, 125–161 (2013)

    MATH  Google Scholar 

  40. Paulavičius, R., Chiter, L., Žilinskas, J.: Global optimization based on bisection of rectangles, function values at diagonals, and a set of Lipschitz constants. J. Glob. Optim. 71(1), 5–20 (2018)

    MathSciNet  MATH  Google Scholar 

  41. Ploskas, N., Sahinidis, N.V.: Review and comparison of algorithms and software for mixed-integer derivative-free optimization. J. Glob. Optim. 82, 433–462 (2022)

    MathSciNet  MATH  Google Scholar 

  42. Powell, M.J.D.: Recent research at Cambridge on radial basis functions. Tech. rep., Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1998)

  43. Powell, M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. In: Di Pillo, G., Roma, M. (eds.) Large-Scale Nonlinear Optimization, pp. 255–297. Springer, New York (2006)

    Google Scholar 

  44. Princeton Library. http://www.gamsworld.org/performance/princetonlib/princetonlib.htm

  45. Regis, R.G., Shoemaker, C.A.: A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19, 497–509 (2007)

    MathSciNet  MATH  Google Scholar 

  46. Regis, R.G., Shoemaker, C.A.: Combining radial basis function surrogates and dynamic coordinate search in high-dimensional expensive black-box optimization. Eng. Optim. 45(5), 529–555 (2013)

    MathSciNet  Google Scholar 

  47. Richtarik, P.: Improved algorithms for convex minimization in relative scale. SIAM J. Optim. 21, 1141–1167 (2011)

    MathSciNet  MATH  Google Scholar 

  48. Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56, 1247–1293 (2013)

    MathSciNet  MATH  Google Scholar 

  49. Schonlau, M.: Computer experiments and global optimization. Ph.D. thesis, Department of Statistics, University of Waterloo, Waterloo, Ontario, Canada (1997)

  50. Sergeyev, Y.D., Kvasov, D.E.: Global search based on efficient diagonal partitions and a set of Lipschitz constants. SIAM J. Optim. 16(3), 910–937 (2006)

    MathSciNet  MATH  Google Scholar 

  51. Sergeyev, Y.D., Kvasov, D.E., Mukhametzhanov, M.S.: On the efficiency of nature-inspired metaheuristics in expensive global optimization with limited budget. Sci. Rep. 8(1), 1–9 (2018)

    Google Scholar 

  52. Sóbester, A., Leary, S.J., Keane, A.J.: On the design of optimization strategies based on global response surface approximation models. J. Glob. Optim. 33, 31–59 (2005)

    MathSciNet  MATH  Google Scholar 

  53. Stripinis, L., Paulavičius, R., Žilinskas, J.: Improved scheme for selection of potentially optimal hyper-rectangles in DIRECT. Optim. Lett. 12(7), 1699–1712 (2018)

    MathSciNet  MATH  Google Scholar 

  54. Tawarmalani, M., Sahinidis, N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103, 225–249 (2005)

    MathSciNet  MATH  Google Scholar 

  55. Torczon, V.J.: On the convergence of pattern search algorithms. SIAM J. Optim. 7, 1–25 (1997)

    MathSciNet  MATH  Google Scholar 

  56. van Beers, A.C., Kleijnen, J.P.C.: Kriging interpolation in simulation: a survey. In: Proceedings of the 2004 Winter Simulation Conference, vol. 1, pp. 121–129 (2004)

  57. Xiao, Y., Rivaz, H., Chabanas, M., Fortin, M., Machado, I., Ou, Y., Heinrich, M.P., Schnabel, J.A., Zhong, X., Maier, A., et al.: Evaluation of MRI to ultrasound registration methods for brain shift correction: the CuRIOUS2018 challenge. IEEE Trans. Med. Imaging 39(3), 777–786 (2019)

    Google Scholar 

  58. Zhai, J., Boukouvala, F.: Data-driven spatial branch-and-bound algorithms for box-constrained simulation-based optimization. J. Glob. Optim. 82, 21–50 (2022)

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA

    Kaiwen Ma

  2. End-to-End Analytics, Lima, Peru

    Luis Miguel Rios

  3. GAMS Development Corporation, Fairfax, VA, USA

    Atharv Bhosekar

  4. H. Milton Stewart School of Industrial and Systems Engineering and School of Chemical and Biomolecular Engineering, Georgia Institute of Technology, Atlanta, GA, USA

    Nikolaos V. Sahinidis

  5. The Dow Chemical Company, Lake Jackson, TX, USA

    Sreekanth Rajagopalan

Authors
  1. Kaiwen Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luis Miguel Rios
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Atharv Bhosekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nikolaos V. Sahinidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sreekanth Rajagopalan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nikolaos V. Sahinidis.

Ethics declarations

Conflict of interest

N. V. Sahinidis declares that he has a financial interest in the ALAMO and BARON software.

Code availability

Interested readers should contact N. V. Sahinidis.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, K., Rios, L.M., Bhosekar, A. et al. Branch-and-Model: a derivative-free global optimization algorithm. Comput Optim Appl 85, 337–367 (2023). https://doi.org/10.1007/s10589-023-00466-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-023-00466-3

Keywords

Search

Navigation