RBFOpt: an open-source library for black-box optimization with costly function evaluations

Abstract

We consider the problem of optimizing an unknown function given as an oracle over a mixed-integer box-constrained set. We assume that the oracle is expensive to evaluate, so that estimating partial derivatives by finite differences is impractical. In the literature, this is typically called a black-box optimization problem with costly evaluation. This paper describes the solution methodology implemented in the open-source library RBFOpt, available on COIN-OR. The algorithm is based on the Radial Basis Function method originally proposed by Gutmann (J Glob Optim 19:201–227, 2001. https://doi.org/10.1023/A:1011255519438), which builds and iteratively refines a surrogate model of the unknown objective function. The two main methodological contributions of this paper are an approach to exploit a noisy but less expensive oracle to accelerate convergence to the optimum of the exact oracle, and the introduction of an automatic model selection phase during the optimization process. Numerical experiments show that RBFOpt is highly competitive on a test set of continuous and mixed-integer nonlinear unconstrained problems taken from the literature: it outperforms the open-source solvers included in our comparison by a large amount, and performs slightly better than a commercial solver. Our empirical evaluation provides insight on which parameterizations of the algorithm are the most effective in practice. The software reviewed as part of this submission was given the Digital Object Identifier (DOI) https://doi.org/10.5281/zenodo.597767.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. 1.

    Achterberg, T.: SCIP: solving constraint integer programs. Math. Program. Comput. 1(1), 1–41 (2009)

    MathSciNet  Article  Google Scholar 

  2. 2.

    Audet, C., Dennis Jr., J.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2004)

    MathSciNet  Article  Google Scholar 

  3. 3.

    Audet, C., Kokkolaras, M., Le Digabel, S., Talgorn, B.: Order-based error for managing ensembles of surrogates in mesh adaptive direct search. J. Glob. Optim. 70(3), 645–675 (2018)

    MathSciNet  Article  Google Scholar 

  4. 4.

    Baudoui, V.: Optimisation robuste multiobjectifs par modèles de substitution. Ph.D. thesis, University of Toulouse Paul Sabatier (2012)

  5. 5.

    Björkman, M., Holmström, K.: Global optimization of costly nonconvex functions using radial basis functions. Optim. Eng. 1(4), 373–397 (2000)

    MathSciNet  Article  Google Scholar 

  6. 6.

    Bonami, P., Biegler, L., Conn, A., Cornuéjols, G., Grossmann, I., Laird, C., Lee, J., Lodi, A., Margot, F., Sawaya, N., Wächter, A.: An algorithmic framework for convex mixed integer nonlinear programs. Discrete Optim. 5, 186–204 (2008)

    MathSciNet  Article  Google Scholar 

  7. 7.

    Bussieck, M.R., Drud, A.S., Meeraus, A.: MINLPLib—a collection of test models for mixed-integer nonlinear programming. INFORMS J. Comput. 15(1), 114–119 (2003)

    MathSciNet  Article  Google Scholar 

  8. 8.

    Byrd, R.H., Nocedal, J., Waltz, R.A.: KNITRO: an integrated package for nonlinear optimization. In: Di Pillo, G., Roma, M. (eds.) Large-Scale Nonlinear Optimization, pp. 35–59. Springer, New York (2006)

    Google Scholar 

  9. 9.

    Conn, A.R., Scheinberg, K., Toint, P.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79(1–3), 397–414 (1997). https://doi.org/10.1007/BF02614326

    MathSciNet  Article  MATH  Google Scholar 

  10. 10.

    Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS-SIAM Series on Optimization. Society for Industrial and Applied Mathematics, Philadelphia (2009)

    Google Scholar 

  11. 11.

    Costa, A., Di Buccio, E., Melucci, M., Nannicini, G.: Efficient parameter estimation for information retrieval using black-box optimization. IEEE Trans. Knowl. Data Eng. 30, 1240–1253 (2017)

    Article  Google Scholar 

  12. 12.

    Costa, A., Nannicini, G., Schroepfer, T., Wortmann, T.: Black-box optimization of lighting simulation in architectural design. In: Cardin, M.A., Krob, D., Chuen, L., Tan, Y., Wood, K. (eds.) Designing Smart Cities: Proceedings of the First Asia-Pacific Conference on Complex Systems Design & Management, CSD&M Asia 2014, pp. 27–39. Springer (2015)

    Google Scholar 

  13. 13.

    D’Ambrosio, C., Nannicini, G., Sartor, G.: MILP models for the selection of a small set of well-distributed points. Oper. Res. Lett. 45(1), 46–52 (2017)

    MathSciNet  Article  Google Scholar 

  14. 14.

    Diaz, G.I., Fokour, A., Nannicini, G., Samulowitz, H.: An effective algorithm for hyperparameter optimization of neural networks. IBM J. Res. Dev. 61(4/5), 9-1 (2017)

    Article  Google Scholar 

  15. 15.

    Dixon, L., Szego, G.: The global optimization problem: an introduction. In: Dixon, L., Szego, G. (eds.) Towards Global Optimization, pp. 1–15. North Holland, Amsterdam (1975)

    Google Scholar 

  16. 16.

    Eriksson, D., Bindel, D., Shoemaker, C.: Surrogate optimization toolbox (pySOT) (2015). http://github.com/dme65/pySOT

  17. 17.

    Fuerle, F., Sienz, J.: Formulation of the Audze–Eglais uniform latin hypercube design of experiments for constrained design spaces. Adv. Eng. Softw. 42(9), 680–689 (2011)

    Article  Google Scholar 

  18. 18.

    Gablonsky, J., Kelley, C.: A locally-biased form of the DIRECT algorithm. J. Glob. Optim. 21(1), 27–37 (2001)

    MathSciNet  Article  Google Scholar 

  19. 19.

    Gendreau, M., Potvin, J.Y. (eds.): Handbook of Metaheuristics, 2nd edn. Kluwer, Dordrecht (2010)

    Google Scholar 

  20. 20.

    Glover, F., Kochenberger, G. (eds.): Handbook of Metaheuristics. Kluwer, Dordrecht (2003)

    Google Scholar 

  21. 21.

    Gomes, C.P., Selman, B., Crato, N., Kautz, H.: Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. J. Autom. Reason. 24(1–2), 67–100 (2000). https://doi.org/10.1023/A:1006314320276

    MathSciNet  Article  MATH  Google Scholar 

  22. 22.

    Gutmann, H.M.: A radial basis function method for global optimization. J. Glob. Optim. 19, 201–227 (2001). https://doi.org/10.1023/A:1011255519438

    MathSciNet  Article  MATH  Google Scholar 

  23. 23.

    Hart, W.E., Laird, C., Watson, J.P., Woodruff, D.L.: Pyomo—optimization Modeling in Python. Springer Optimization and Its Applications, vol. 67. Springer, Berlin (2012)

    Google Scholar 

  24. 24.

    Hart, W.E., Watson, J.P., Woodruff, D.L.: Pyomo: modeling and solving mathematical programs in Python. Math. Program. Comput. 3(3), 219–260 (2011). https://doi.org/10.1007/s12532-011-0026-8

    MathSciNet  Article  Google Scholar 

  25. 25.

    Hemker, T.: Derivative free surrogate optimization for mixed-integer nonlinear black-box problems in engineering. Master’s thesis, Technischen Universität Darmstadt (2008)

  26. 26.

    Holmström, K.: An adaptive radial basis algorithm (ARBF) for expensive black-box global optimization. J. Glob. Optim. 41(3), 447–464 (2008)

    MathSciNet  Article  Google Scholar 

  27. 27.

    Holmström, K., Quttineh, N.H., Edvall, M.M.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization. Optim. Eng. 9(4), 311–339 (2008)

    MathSciNet  Article  Google Scholar 

  28. 28.

    Huyer, W., Neumaier, A.: SNOBFIT—stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35(2), 1–25 (2008)

    MathSciNet  Article  Google Scholar 

  29. 29.

    Ilievski, I., Akhtar, T., Feng, J., Shoemaker, C.A.: Efficient hyperparameter optimization of deep learning algorithms using deterministic RBF surrogates. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

  30. 30.

    Jakobsson, S., Patriksson, M., Rudholm, J., Wojciechowski, A.: A method for simulation based optimization using radial basis functions. Optim. Eng. 11(4), 501–532 (2010)

    MathSciNet  Article  Google Scholar 

  31. 31.

    Johnson, S.G.: The NLopt nonlinear-optimization package. http://ab-initio.mit.edu/nlopt

  32. 32.

    Jones, D., Perttunen, C., Stuckman, B.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79(1), 157–181 (1993)

    MathSciNet  Article  Google Scholar 

  33. 33.

    Jones, D., Schonlau, M., Welch, W.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)

    MathSciNet  Article  Google Scholar 

  34. 34.

    Kolda, T.G., Lewis, R.M., Torczon, V.J.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)

    MathSciNet  Article  Google Scholar 

  35. 35.

    Le Digabel, S.: Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37(4), 44:1–44:15 (2011). https://doi.org/10.1145/1916461.1916468

    MathSciNet  Article  MATH  Google Scholar 

  36. 36.

    MINLP Library 2. http://www.gamsworld.org/minlp/minlplib2/html/

  37. 37.

    Moré, J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)

    MathSciNet  Article  Google Scholar 

  38. 38.

    Müller, J.: MISO: mixed-integer surrogate optimization framework. Optim. Eng. 1–27 (2015). https://doi.org/10.1007/s11081-015-9281-2

    MathSciNet  Article  Google Scholar 

  39. 39.

    Müller, J., Paudel, R., Shoemaker, C.A., Woodbury, J., Wang, Y., Mahowald, N.: \(\text{ CH }_{4}\) parameter estimation in CLM4.5bgc using surrogate global optimization. Geosci. Model Dev. 8(10), 3285–3310 (2015). https://doi.org/10.5194/gmd-8-3285-2015

    Article  Google Scholar 

  40. 40.

    Müller, J., Shoemaker, C.A.: Influence of ensemble surrogate models and sampling strategy on the solution quality of algorithms forcomputationally expensive black-box global optimization problems. J. Glob. Optim. 60(2), 123–144 (2014). https://doi.org/10.1007/s10898-014-0184-0

    Article  MATH  Google Scholar 

  41. 41.

    Müller, J., Shoemaker, C.A., Piché, R.: SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40(5), 1383–1400 (2013). https://doi.org/10.1016/j.cor.2012.08.022

    MathSciNet  Article  MATH  Google Scholar 

  42. 42.

    Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965)

    MathSciNet  Article  Google Scholar 

  43. 43.

    Neumaier, A.: Neumaier’s collection of test problems for global optimization. http://www.mat.univie.ac.at/~neum/glopt/my_problems.html. Retrieved in May 2014

  44. 44.

    Powell, M.: Recent research at Cambridge on radial basis functions. In: Müller, M.W., Buhmann, M.D., Mache, D.H., Felten, M. (eds.) New Developments in Approximation Theory. International Series of Numerical Mathematics, vol. 132, pp. 215–232. Birkhauser Verlag, Basel (1999)

    Google Scholar 

  45. 45.

    Powell, M.J.: The BOBYQA algorithm for bound constrained optimization without derivatives. Technical Report, Cambridge NA Report NA2009/06, University of Cambridge (2009)

  46. 46.

    Regis, R., Shoemaker, C.: Improved strategies for radial basis function methods for global optimization. J. Glob. Optim. 37, 113–135 (2007). https://doi.org/10.1007/s10898-006-9040-1

    MathSciNet  Article  MATH  Google Scholar 

  47. 47.

    Regis, R.G., Shoemaker, C.A.: A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19(4), 497–509 (2007). https://doi.org/10.1287/ijoc.1060.0182

    MathSciNet  Article  MATH  Google Scholar 

  48. 48.

    Regis, R.G., Shoemaker, C.A.: A quasi-multistart framework for global optimization of expensive functions using response surface models. J. Glob. Optim. 56(4), 1719–1753 (2013)

    MathSciNet  Article  Google Scholar 

  49. 49.

    Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)

    MathSciNet  Article  Google Scholar 

  50. 50.

    Schoen, F.: A wide class of test functions for global optimization. J. Glob. Optim. 3(2), 133–137 (1993)

    MathSciNet  Article  Google Scholar 

  51. 51.

    Törn, A., Žilinskas, A.: Global Optimization. Springer, Berlin (1987)

    Google Scholar 

  52. 52.

    Wächter, A., Biegler, L.T.: On the implementation of a primal-dual interior point filter line search algorithm for large-scale nonlinear programming. Math. Program. 106(1), 25–57 (2006)

    MathSciNet  Article  Google Scholar 

  53. 53.

    Wortmann, T., Costa, A., Nannicini, G., Schroepfer, T.: Advantages of surrogate models for architectural design optimization. Artif. Intell. Eng. Des. Anal. Manuf. 29(4), 471–481 (2015)

    Article  Google Scholar 

  54. 54.

    Wortmann, T., Waibel, C., Nannicini, G., Evins, R., Schroepfer, T., Carmeliet, J.: Are genetic algorithms really the best choice for building energy optimization? In: Proceedings of the Symposium on Simulation for Architecture & Urban Design (SimAUD), pp. 51–58. SCS, Toronto, Canada (2017)

Download references

Acknowledgements

The authors are grateful for partial support by the SUTD-MIT International Design Center under grant IDG21300102. The research of A. C. was partially conducted at the Future Resilient Systems and the Future Cities Laboratory at the Singapore-ETH Centre (SEC). The SEC was established as a collaboration between ETH Zurich and National Research Foundation (NRF) Singapore (FI 370074011, FI 370074016) under the auspices of the NRF’s Campus for Research Excellence and Technological Enterprise (CREATE) program.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Giacomo Nannicini.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Costa, A., Nannicini, G. RBFOpt: an open-source library for black-box optimization with costly function evaluations. Math. Prog. Comp. 10, 597–629 (2018). https://doi.org/10.1007/s12532-018-0144-7

Download citation

Keywords

  • Black-box optimization
  • Derivative-free optimization
  • Global optimization
  • Radial basis function
  • Open-source software
  • Mixed-integer nonlinear programming

Mathematics Subject Classification

  • 90C56
  • 90C30
  • 65K05
  • 97N80