Journal of Global Optimization

, Volume 13, Issue 4, pp 455–492 | Cite as

Efficient Global Optimization of Expensive Black-Box Functions

  • Donald R. Jones
  • Matthias Schonlau
  • William J. Welch


In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.

Bayesian global optimization Kriging Random function Response surface Stochastic process Visualization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alexandrov, N. M., Dennis, Jr., J. E., Lewis, R. M. and Torczon, V. (1998), A trust-region framework for managing the use of approximation models in optimization, Structural Optimization 15: 16–23.Google Scholar
  2. 2.
    Androulakis, I. P., Maranas, C. D. and Floudas, C. A. (1995), αBB: a global optimization method for general constrained nonconvex problems, Journal of Global Optimization 7: 337–363.Google Scholar
  3. 3.
    Aslett, R., Buck, R. J., Duvall, S. G., Sacks, J. and Welch, W. J. (1998), Circuit optimization via sequential computer experiments: design of an output buffer, Applied Statistics 47: 31–48.Google Scholar
  4. 4.
    Betro, B. (1991), Bayesian methods in global optimization, Journal of Global Optimization 1: 1–14.Google Scholar
  5. 5.
    Boender, C. G. E. and Romeijn, H. E. (1995), Stochastic methods, in R. Horst and P. M. Pardalos, eds., Handbook of Global Optimization, pp. 829–869, Kluwer Academic Publishers, Dordrecht/Boston/London.Google Scholar
  6. 6.
    Booker, A. J., Conn, A. R., Dennis, J. E., Frank, P. D., Trossett, M. and Torczon, V. (1995), Global modeling for optimization, Boeing Information and Support Services, Technical Report ISSTECH–95-032.Google Scholar
  7. 7.
    Booker, A. J., Dennis, J. E., Frank, P. D., Serafini, D. B. and Torczon, V. (1997), Optimization using surrogate objectives on a helicopter test example, Boeing Shared Services Group, Technical Report SSGTECH–97-027.Google Scholar
  8. 8.
    Box, G. E. P., Hunter, W. G. and Hunter, J. S. (1978), Statistics for Experimenters, John Wiley, New York.Google Scholar
  9. 9.
    Cox, D. D. and John. S. (1997), SDO: A statistical method for global optimization, in N. Alexandrov and M. Y. Hussaini, eds., Multidisciplinary Design Optimization: State of the Art, pp. 315–329, SIAM, Philadelphia.Google Scholar
  10. 10.
    Cressie. N. (1989), Geostatistics, The American Statistician 43: 197–202. See also the comment on this article by G. Wahba and the reply by N. Cressie (1990), in The American Statistician 44: 255–258.Google Scholar
  11. 11.
    Cressie, N. (1990), The origins of kriging, Mathematical Geology 22: 239–252.Google Scholar
  12. 12.
    Cressie, N. (1993), Statistics for Spatial Data, John Wiley, New York.Google Scholar
  13. 13.
    Currin, C., Mitchell, T., Morris, M. and Ylvisaker, D. (1991), Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments, Journal of the American Statistical Association 86: 953–963.Google Scholar
  14. 14.
    Dixon, L. C. W. and Szego, G. P. (1978), The global optimisation problem: an introduction, in L. C. W. Dixon and G. P. Szego (eds.), Towards Global Optimisation, Vol. 2, pp. 1–15. North Holland, Amsterdam.Google Scholar
  15. 15.
    Eby, D., Averill, R. C., Punch III, W. F. and Goodman, E. D. (1998), Evaluation of injection island GA performance on flywheel design optimization, in I. C. Parmee (ed.), Adaptive Computing in Design and Manufacture, Springer Verlag.Google Scholar
  16. 16.
    Elder IV, J. F. (1992), Global R d optimization when probes are expensive: the GROPE algorithm. Proceedings of the 1992 IEEE International Conference on Systems, Man, and Cybernetics, Vol. 1, pp. 577–582, Chicago.Google Scholar
  17. 17.
    Gao, F., Sacks, J. and Welch, W. J. (1996), Predicting urban ozone levels and trends with semiparametric modeling, Journal of Agricultural, Biological, and Environmental Statistics 1: 404–425.Google Scholar
  18. 18.
    Koehler, J. and Owen, A. (1996), Computer experiments, in S. Ghosh and C. R. Rao (eds.), Handbook of Statistics, 13: Design and Analysis of Experiments, pp. 261–308, Elsevier, Amsterdam.Google Scholar
  19. 19.
    Kushner, H. J. (1964), A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise, Journal of Basic Engineering 86: 97–106.Google Scholar
  20. 20.
    Locatelli, M. (1997), Bayesian algorithms for one-dimensional global optimization, Journal of Global Optimization 10: 57–76.Google Scholar
  21. 21.
    Matheron, G. (1963), Principles of geostatistics, Economic Geology 58: 1246–1266.Google Scholar
  22. 22.
    McKay, M. D., Conover, W. J. and Beckman, R. J. (1979), A comparison of three methods for selecting values of input variables in the analysis of output from a computer code, Technometrics 21: 239–245.Google Scholar
  23. 23.
    Mockus, J. (1994), Application of Bayesian approach to numerical methods of global and stochastic optimization, Journal of Global Optimization 4: 347–365.Google Scholar
  24. 24.
    Mockus, J., Tiesis, V. and Zilinskas, A. (1978), The application of Bayesian methods for seeking the extremum, in L. C. W. Dixon and G. P. Szego (eds.), Towards Global Optimisation, Vol.2, pp. 117–129. North Holland, Amsterdam.Google Scholar
  25. 25.
    Morris, M. D., Mitchell, T. J. and Ylvisaker, D. (1993), Bayesian design and analysis of computer experiments: use of derivatives in surface prediction, Technometrics 35: 243–255.Google Scholar
  26. 26.
    Parzen, E. (1963), A new approach to the synthesis of optimal smoothing and prediction systems, in R. Bellman, (ed.), Mathematical Optimization Techniques, pp. 75–108, University of California Press, Berkeley.Google Scholar
  27. 27.
    Perttunen, C. (1991), A computational geometric approach to feasible region division in constrained global optimization, Proceedings of the 1991 IEEE Conference on Systems, Man, and Cybernetics, Vol. 1, pp. 585–590.Google Scholar
  28. 28.
    Press, W. H., Flannery, B. P., Teukolsky, S. A. and Vetterling, W. T. (1993), Numerical Recipes in FORTRAN, Cambridge University Press.Google Scholar
  29. 29.
    Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P. (1989), Design and analysis of computer experiments (with discussion), Statistical Science 4: 409–435.Google Scholar
  30. 30.
    Sacks, J. and Ylvisaker, D. (1970), Statistical designs and integral approximation, in R. Pyke (ed.), Proceedings of the Twelfth Biennial Seminar of the Canadian Mathematical Congress, pp. 115–136, Canadian Mathematical Congress, Montreal.Google Scholar
  31. 31.
    Schonlau, M. (1997), Computer experiments and global optimization, Ph. D. Thesis, University of Waterloo, Waterloo, Ontario, Canada.Google Scholar
  32. 32.
    Schonlau, M., Welch, W. J. and Jones, D. R. (1998), Global versus local search in constrained optimization of computer models, to appear in N. Flournoy, W. F. Rosenberger and W. K. Wong (eds.), New Developments and Applications in Experimental Design, Institute of Mathematical Statistics. Also available as Technical Report RR–97-11, Institute for Improvement in Quality and Productivity, University of Waterloo, Waterloo, Ontario, Canada, December 1997.Google Scholar
  33. 33.
    Stuckman, B. E. (1988), A global search method for optimizing nonlinear systems, IEEE Transactions on Systems, Man, and Cybernetics 18: 965–977.Google Scholar
  34. 34.
    Theil, H. (1971), Principles of Econometrics, John Wiley, New York.Google Scholar
  35. 35.
    Ver Hoef, J. M. and Cressie, N. (1993), Multivariable spatial prediction, Mathematical Geology 25: 219–240.Google Scholar
  36. 36.
    Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J. and Morris, M. D. (1992), Screening, predicting, and computer experiments, Technometrics 34: 15–25.Google Scholar
  37. 37.
    Zilinskas, A. (1992), A review of statistical models for global optimization, Journal of Global Optimization 2: 145–153.Google Scholar

Copyright information

© Kluwer Academic Publishers 1998

Authors and Affiliations

  • Donald R. Jones
    • 1
  • Matthias Schonlau
    • 2
  • William J. Welch
    • 3
  1. 1.Operations Research DepartmentGeneral Motors R&D OperationsWarrenUSA
  2. 2.National Institute of Statistical Sciences, Research Triangle ParkUSA
  3. 3.Department of Statistics and Actuarial Science and The Institute for Improvement in Quality and ProductivityUniversity of WaterlooWaterlooCanada

Personalised recommendations