Kriging Is Well-Suited to Parallelize Optimization

  • David Ginsbourger
  • Rodolphe Le Riche
  • Laurent Carraro
Part of the Adaptation Learning and Optimization book series (ALO, volume 2)


The optimization of expensive-to-evaluate functions generally relies on metamodel-based exploration strategies. Many deterministic global optimization algorithms used in the field of computer experiments are based on Kriging (Gaussian process regression). Starting with a spatial predictor including a measure of uncertainty, they proceed by iteratively choosing the point maximizing a criterion which is a compromise between predicted performance and uncertainty. Distributing the evaluation of such numerically expensive objective functions on many processors is an appealing idea. Here we investigate a multi-points optimization criterion, the multipoints expected improvement (\(q-{\mathbb E}I\)), aimed at choosing several points at the same time. An analytical expression of the \(q-{\mathbb E}I\) is given when q = 2, and a consistent statistical estimate is given for the general case. We then propose two classes of heuristic strategies meant to approximately optimize the \(q-{\mathbb E}I\), and apply them to the classical Branin-Hoo test-case function. It is finally demonstrated within the covered example that the latter strategies perform as good as the best Latin Hypercubes and Uniform Designs ever found by simulation (2000 designs drawn at random for every q ∈ [1,10]).


Ordinary Kriging Kriging Model Heuristic Strategy Expect Improvement Gaussian Process Regression 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abrahamsen, P.: A review of gaussian random fields and correlation functions, 2nd edn. Tech. Rep. 917, Norwegian Computing Center, Olso (1997)Google Scholar
  2. 2.
    Antoniadis, A., Berruyer, J., Carmona, R.: Régression non linéaire et applications. Economica, Paris (1992)Google Scholar
  3. 3.
    Baker, C., Watson, L.T., Grossman, B., Mason, W.H., Haftka, R.T.: Parallel global aircraft configuration design space exploration. Practical parallel computing, 79–96 (2001)Google Scholar
  4. 4.
    Bishop, C.: Neural Networks for Pattern Recognition. Oxford Univ. Press, Oxford (1995)Google Scholar
  5. 5.
    Blum, C.: Ant colony optimization: introduction and recent trends. Physics of Life Review 2, 353–373 (2005)CrossRefGoogle Scholar
  6. 6.
    development Core Team R: R: A language and environment for statistical computing (2006),
  7. 7.
    Cressie, N.: Statistics for spatial data. Wiley series in probability and mathematical statistics (1993)Google Scholar
  8. 8.
    Dreyfus, G., Martinez, J.M.: Réseaux de neurones. Eyrolles (2002)Google Scholar
  9. 9.
    Eiben, A., Smith, J.: Introduction to Evolutionary Computing. Springer, Heidelberg (2003)zbMATHGoogle Scholar
  10. 10.
    Emmerich, M., Giannakoglou, K., Naujoks, B.: Single-and multiobjective optimization assisted by gaussian random field metamodels. IEEE Transactions on Evolutionary Computation 10(4), 421–439 (2006)CrossRefGoogle Scholar
  11. 11.
    Geman, D., Jedynak, B.: An active testing model for tracking roads in satellite images. Tech. rep., Institut National de Recherches en Informatique et Automatique (INRIA) (December 1995)Google Scholar
  12. 12.
    Genton, M.: Classes of kernels for machine learning: A statistics perspective. Journal of Machine Learning Research 2, 299–312 (2001)CrossRefGoogle Scholar
  13. 13.
    Ginsbourger, D.: Multiples métamodèles pour l’approximation et l’optimisation de fonctions numériques multivariables. PhD thesis, Ecole Nationale Supérieure des Mines de Saint-Etienne (2009)Google Scholar
  14. 14.
    Ginsbourger, D., Le Riche, R., Carraro, L.: A multipoints criterion for parallel global optimization of deterministic computer experiments. In: Non-Convex Programming 2007 (2007)Google Scholar
  15. 15.
    Goria, S.: Evaluation d’un projet minier: approche bayésienne et options réelles. PhD thesis, Ecole des Mines de Paris (2004)Google Scholar
  16. 16.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, Heidelberg (2001)zbMATHGoogle Scholar
  17. 17.
    Henkenjohann, N., Göbel, R., Kleiner, M., Kunert, J.: An adaptive sequential procedure for efficient optimization of the sheet metal spinning process. Qual. Reliab. Engng. Int. 21, 439–455 (2005)CrossRefGoogle Scholar
  18. 18.
    Huang, D., Allen, T., Notz, W., Miller, R.: Sequential Kriging optimization using multiple fidelity evaluations. Sructural and Multidisciplinary Optimization 32, 369–382 (2006)CrossRefGoogle Scholar
  19. 19.
    Huang, D., Allen, T., Notz, W., Zheng, N.: Global optimization of stochastic black-box systems via sequential Kriging meta-models. Journal of Global Optimization 34, 441–466 (2006)zbMATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Jones, D.: A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization 21, 345–383 (2001)zbMATHCrossRefGoogle Scholar
  21. 21.
    Jones, D., Pertunen, C., Stuckman, B.: Lipschitzian optimization without the lipschitz constant. Journal of Optimization Theory and Application 79(1) (1993)Google Scholar
  22. 22.
    Jones, D., Schonlau, M., Welch, W.: Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13, 455–492 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  23. 23.
    Journel, A.: Fundamentals of geostatistics in five lessons. Tech. rep., Stanford Center for Reservoir Forecasting (1988)Google Scholar
  24. 24.
    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: IEEE Intl. Conf. on Neural Networks, vol. 4, pp. 1942–1948 (1995)Google Scholar
  25. 25.
    Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983)CrossRefMathSciNetGoogle Scholar
  26. 26.
    Knowles, J.: Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionnary Computation (2005)Google Scholar
  27. 27.
    Koehler, J., Owen, A.: Computer experiments. Tech. rep., Department of Statistics, Stanford University (1996)Google Scholar
  28. 28.
    Kracker, H.: Methoden zur analyse von computerexperimenten mit anwendung auf die hochdruckblechumformung. Master’s thesis, Dortmund University (2006)Google Scholar
  29. 29.
    Krige, D.: A statistical approach to some basic mine valuation problems on the witwatersrand. J. of the Chem., Metal. and Mining Soc. of South Africa 52(6), 119–139 (1951)Google Scholar
  30. 30.
    Martin, J., Simpson, T.: A monte carlo simulation of the Kriging model. In: 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Albany, NY, AIAA, AIAA-2004-4483, August 30 - September 2 (2004)Google Scholar
  31. 31.
    Martin, J., Simpson, T.: Use of Kriging models to approximate deterministic computer models. AIAA Journal 43(4), 853–863 (2005)CrossRefGoogle Scholar
  32. 32.
    Matheron, G.: Principles of geostatistics. Economic Geology 58, 1246–1266 (1963)CrossRefGoogle Scholar
  33. 33.
    Matheron, G.: La théorie des variables régionalisées et ses applications. Tech. rep., Centre de Morphologie Mathématique de Fontainebleau, Ecole Nationale Supérieure des Mines de Paris (1970)Google Scholar
  34. 34.
    O’Hagan, A.: Bayesian analysis of computer code outputs: a tutorial. Reliability Engineering and System Safety 91(91), 1290–1300 (2006)CrossRefGoogle Scholar
  35. 35.
    Paciorek, C.: Nonstationary gaussian processes for regression and spatial modelling. PhD thesis, Carnegie Mellon University (2003)Google Scholar
  36. 36.
    Ponweiser, W., Wagner, T., Biermann, D., Vincze, M.: Multiobjective optimization on a limited budget of evaluations using model-assisted S-metric selection. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 784–794. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  37. 37.
    Praveen, C., Duvigneau, R.: Radial basis functions and Kriging metamodels for aerodynamic optimization. Tech. rep., INRIA (2007)Google Scholar
  38. 38.
    Queipo, N., Verde, A., Pintos, S., Haftka, R.: Assessing the value of another cycle in surrogate-based optimization. In: 11th Multidisciplinary Analysis and Optimization Conference, AIAA (2006)Google Scholar
  39. 39.
    Rasmussen, C., Williams, K.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006)zbMATHGoogle Scholar
  40. 40.
    Ripley, B.: Stochastic Simulation. John Wiley and Sons, New York (1987)zbMATHCrossRefGoogle Scholar
  41. 41.
    Roustant, O., Ginsbourger, D., Deville, Y.: The DiceKriging package: Kriging-based metamodeling and optimization for computer experiments. In: The UseR! Conference, Agrocampus-Ouest, Rennes, France (2009)Google Scholar
  42. 42.
    Sacks, J., Welch, W., Mitchell, T., Wynn, H.: Design and analysis of computer experiments. Statistical Science 4(4), 409–435 (1989)zbMATHCrossRefMathSciNetGoogle Scholar
  43. 43.
    Santner, T., Williams, B., Notz, W.: The Design and Analysis of Computer Experiments. Springer, Heidelberg (2003)zbMATHGoogle Scholar
  44. 44.
    Sasena, M.: Flexibility and efficiency enhancements for constrained global design optimization with Kriging approximations. PhD thesis, University of Michigan (2002)Google Scholar
  45. 45.
    Sasena, M.J., Papalambros, P., Goovaerts, P.: Exploration of metamodeling sampling criteria for constrained global optimization. Journal of Engineering Optimization (2002)Google Scholar
  46. 46.
    Sasena, M.J., Papalambros, P.Y., Goovaerts, P.: Global optimization of problems with disconnected feasible regions via surrogate modeling. In: Proceedings of the 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Atlanta, GA (2002)Google Scholar
  47. 47.
    Schonlau, M.: Computer experiments and global optimization. PhD thesis, University of Waterloo (1997)Google Scholar
  48. 48.
    Schonlau, M., Welch, W., Jones, D.: A data-analytic approach to bayesian global optimization. In: Proceedings of the A.S.A. (1997)Google Scholar
  49. 49.
    Ulmer, H., Streichert, F., Zell, A.: Evolution strategies assisted by gaussian processes with improved pre-selection criterion. Tech. rep., Center for Bioinformatics Tuebingen, ZBIT (2003)Google Scholar
  50. 50.
    Villemonteix, J.: Optimisation de fonctions coûteuses: Modèles gaussiens pour une utilisation théorie et pratique industrielle. PhD thesis, Université Paris-sud XI, Faculté des Sciences d’Orsay (2008)Google Scholar
  51. 51.
    Villemonteix, J., Vazquez, E., Walter, E.: An informational approach to the global optimization of expensive-to-evaluate functions. Journal of Global Optimization 44(4), 509–534 (2009)zbMATHCrossRefMathSciNetGoogle Scholar
  52. 52.
    Wahba, G.: Spline Models for Observational Data. SIAM, Philadelphia (1990)zbMATHGoogle Scholar
  53. 53.
    Williams, C., Rasmussen, C.: Gaussian processes for regression. In: Advances in Neural Information Processing Systems, vol. 8 (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • David Ginsbourger
    • 1
  • Rodolphe Le Riche
    • 2
  • Laurent Carraro
    • 1
  1. 1.Département 3MI, Ecole Nationale Supérieure des MinesSaint-EtienneFrance
  2. 2.CNRS (UMR 5146) and Département 3MI, Ecole Nationale Supérieure des MinesSaint-EtienneFrance

Personalised recommendations