Global Optimization with Sparse and Local Gaussian Process Models

  • Tipaluck KrityakierneEmail author
  • David Ginsbourger
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9432)


We present a novel surrogate model-based global optimization framework allowing a large number of function evaluations. The method, called SpLEGO, is based on a multi-scale expected improvement (EI) framework relying on both sparse and local Gaussian process (GP) models. First, a bi-objective approach relying on a global sparse GP model is used to determine potential next sampling regions. Local GP models are then constructed within each selected region. The method subsequently employs the standard expected improvement criterion to deal with the exploration-exploitation trade-off within selected local models, leading to a decision on where to perform the next function evaluation(s). The potential of our approach is demonstrated using the so-called Sparse Pseudo-input GP as a global model. The algorithm is tested on four benchmark problems, whose number of starting points ranges from \(10^{2}\) to \(10^{4}\). Our results show that SpLEGO is effective and capable of solving problems with large number of starting points, and it even provides significant advantages when compared with state-of-the-art EI algorithms.


Black-box optimization Expected improvement Kriging 


  1. 1.
    Ackley, D.H.: A Connectionist Machine for Genetic Hillclimbing. Kluwer, Dordrecht (1987)CrossRefGoogle Scholar
  2. 2.
    Chevalier, C., Ginsbourger, D.: Fast computation of the multipoint expected improvement with applications in batch selection. In: Giuseppe, N., Panos, P. (eds.) Learning and Intelligent Optimization, pp. 59–69. Springer, Heidelberg (2014)Google Scholar
  3. 3.
    Csató, L., Opper, M.: Sparse on-line gaussian processes. Neural Comput. 14(3), 641–668 (2002)zbMATHCrossRefGoogle Scholar
  4. 4.
    Dixon, L.C.W., Szegö, G.P.: The global optimization problem: an introduction. Towards Glob. Optim. 2, 1–15 (1978)Google Scholar
  5. 5.
    Forrester, A.I.J., Keane, A.J.: Recent advances in surrogate-based optimization. Progr. Aerosp. Sci. 45(1), 50–79 (2009)CrossRefGoogle Scholar
  6. 6.
    Hansen, N., Finck, S., Ros, R., Auger, A., et al.: Real-parameter black-box optimization benchmarking 2009: noiseless functions definitions (2009)Google Scholar
  7. 7.
    Janusevskis, J., Le Riche, R., Ginsbourger, D., Girdziusas, R.: Expected improvements for the asynchronous parallel global optimization of expensive functions: potentials and challenges. In: Hamadi, Y., Schoenauer, M. (eds.) LION 2012. LNCS, vol. 7219, pp. 413–418. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  8. 8.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)zbMATHMathSciNetCrossRefGoogle Scholar
  9. 9.
    Mockus, J.: Bayesian approach to global optimization. Springer, The Netherlands (1989)zbMATHCrossRefGoogle Scholar
  10. 10.
    Myers, R.H., Anderson-Cook, C.M.: Response Surface Methodology: Process and Product Optimization using Designed Experiments, vol. 705. Wiley, New York (2009)Google Scholar
  11. 11.
    Quiñonero-Candela, J., Rasmussen, C.E.: A unifying view of sparse approximate gaussian process regression. J. Mach. Learn. Res. 6, 1939–1959 (2005)zbMATHMathSciNetGoogle Scholar
  12. 12.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press, Cambridge (2006)zbMATHGoogle Scholar
  13. 13.
    Regis, R.G., Shoemaker, C.A.: A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19(4), 497–509 (2007)zbMATHMathSciNetCrossRefGoogle Scholar
  14. 14.
    Roustant, O., Ginsbourger, D., Deville, Y.: DiceKriging, DiceOptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. J. Stat. Softw. 51, 1–55 (2012)CrossRefGoogle Scholar
  15. 15.
    Smola, A.J., Bartlett, P.: Sparse greedy gaussian process regression. In: Advances in Neural Information Processing Systems, vol. 13. Citeseer (2001)Google Scholar
  16. 16.
    Snelson, E., Ghahramani, Z.: Sparse gaussian processes using pseudo-inputs. In: Schölkopf, B., Weiss, Y., Platt, J. (eds.) Advances in Neural Information Processing Systems, vol. 18. MIT Press, Cambridge (2006)Google Scholar
  17. 17.
    Snelson, E., Ghahramani, Z.: Variable noise and dimensionality reduction for sparse gaussian processes. In: Proceedings of the 22nd International Conference on Uncertainty in Artificial Intelligence (2006)Google Scholar
  18. 18.
    Snoek, J., et al.: Scalable bayesian optimization using deep neural networks (2015). arXiv preprint arXiv:1502.05700
  19. 19.
    Sóbester, A., Leary, S.J., Keane, A.J.: On the design of optimization strategies based on global response surface approximation models. J. Glob. Optim. 33(1), 31–59 (2005)zbMATHCrossRefGoogle Scholar
  20. 20.
    Stein, M.L.: Interpolation of Spatial Data: Some Theory for Kriging. Springer, New York (1999)zbMATHCrossRefGoogle Scholar
  21. 21.
    Titsias, M.K.: Variational learning of inducing variables in sparse gaussian processes. In: International Conference on Artificial Intelligence and Statistics, pp. 567–574 (2009)Google Scholar
  22. 22.
    Veenendaal, G.V.: Tree-GP: a scalable bayesian global numerical optimization algorithm. Master’s thesis, Utrecht University, The Netherlands (2015)Google Scholar
  23. 23.
    Wang, G.G., Shan, S.: Review of metamodeling techniques in support of engineering design optimization. J. Mech. Des. 129(4), 370–380 (2007)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Department of Mathematics and Statistics, IMSVUniversity of BernBernSwitzerland
  2. 2.Idiap Research InstituteMartignySwitzerland

Personalised recommendations