Skip to main content

A rigorous framework for optimization of expensive functions by surrogates

Abstract

The goal of the research reported here is to develop rigorous optimization algorithms to apply to some engineering design problems for which direct application of traditional optimization approaches is not practical. This paper presents and analyzes a framework for generating a sequence of approximations to the objective function and managing the use of these approximations as surrogates for optimization. The result is to obtain convergence to a minimizer of an expensive objective function subject to simple constraints. The approach is widely applicable because it does not require, or even explicitly approximate, derivatives of the objective. Numerical results are presented for a 31-variable helicopter rotor blade design example and for a standard optimization test example.

This is a preview of subscription content, access via your institution.

References

  1. Barthelemy, J-F.M.; Haftka, R.T. 1993: Approximation concepts for optimum structural design — a review.Struct. Optim. 5, 129–144

    Google Scholar 

  2. Booker, A.J. 1994: DOE for computer output.Technical Report BCSTECH-94-052 Boeing Computer Services, Research and Technology, M/S 7L-68, Seattle, Washington 98124

    Google Scholar 

  3. Booker, A.J. 1996: Case studies in design and analysis of computer experiments. InProc. Section on Physical and Engineering Sciences, American Statistical Association

  4. Booker, A.J.; Conn, A.R., Dennis, J.E. Jr.; Frank, P.D.; Trosset, M.W.; Torczon, V. 1995. Global modeling for optimization Boeing/IBM/Rice collaborative project 1995 final report.Technical Report ISSTECH-95-032, Boeing Information & Support Services, Research and Technology, M/S 7L-68, Seattle, Washington 98124

    Google Scholar 

  5. Booker, A.J.; Dennis, J.E., Jr.; Frank, P.D.; Serafini, D.B.; Torczon, V. 1997. Optimization using surrogate objectives on a helicopter test example.Technical Report SSGTECH-97-027, Roeing Shared Services Group, Applied Research & Technology, M/S 7L-68, Seattle, Washington 98124. Also available as Technical Report 97-31, Department of Computational and Applied Mathematics, Rice University, Houston, Texas 77005-1892. To appear in: Borggaard, J.; Burns, J.; Cliff, E.; Schreck, S. (eds.)Optimal design and control. Cambridge, MA: Birkhauser

    Google Scholar 

  6. Burgee, S.L.; Giunta, A.A.; Balabanov, V.; Grossman, B.; Mason, W.H.; Narducci, R.; Haftka, R.T.; Watson, L.T. 1996: A coarse-grained parallel variable-complexity multidisciplinary optimization paradigm.Int. J. Supercomputing Appl. & High Performance Computing 10, 269–299

    Google Scholar 

  7. Conn, A.R.; Scheinberg, K.; Toint, Ph.L. 1997: On the convergence of derivative-free methods for unconstrained optimization. In: Iserles, A.; Buhmann, M. (eds.)Approximation theory and optimization: tributes to M.J.D. Powell, pp. 83–108. Cambridge: Cambridge University Press

    Google Scholar 

  8. Conn, A.R.; Toint, Ph.L. 1996: An algorithm using quadratic interpolation for unconstrained derivative free optimization. In: Di Pillo, G.; Giannessi, F. (eds.)Nonlinear optimization and applications, pp. 27–47. New York: Plenum Publishing

    Google Scholar 

  9. Cox, D.D.; John, S. 1997: SDO: a statistical method for global optimization. In: Alexandrov, N.; Hussaini, M.Y. (eds.)Multidisciplinary design optimization: state of the art, pp. 315–329. Philadelphia: SIAM

    Google Scholar 

  10. Currin, C.; Mitchell, T.; Morris, M.; Ylvisaker, D. 1988: A Bayesian approach to the design and analysis of computer experiments.Technical Report ORNL-6498, Oak Ridge National Laboratory

  11. Davis, C. 1954: Theory of positive linear dependence.Amer. J. Math. 76, 733–746

    Google Scholar 

  12. De Boor, C.; Ron, A. 1992: Computational aspects of polynomial interpolation in several variables.Mathematics of Computation 58, 705–727

    Google Scholar 

  13. Dennis, J.E., Jr.; Torczon, V. 1991: Direct search methods on parallel machines.SIAM J. Optimiz. 1, 448–474

    Google Scholar 

  14. Dennis, J.E., Jr.; Torczon, V. 1997: Managing approximation models in optimization. In: Alexandrov, N.; Hussaini, M.Y. (eds.)Multidisciplinary design optimization: state-of-the-art, pp. 330–347. Philadelphia: SIAM. Also available as Technical Report 95-19, Department of Computational and Applied Mathematics, Rice University, Houston, Texas 77005-1892

    Google Scholar 

  15. Dennis, J.E., Jr.; Walker, H.F. 1984: Inaccuracy in quasi-Newton methods: local improvement theorems.Math. Prog. Study 22, 70–85

    Google Scholar 

  16. Dixon, L.C.W.; Szegö G.P. (eds.) 1978:Towards global optimization 2. Amsterdam: North-Holland Pub. Co.

    Google Scholar 

  17. Efron, B.; Stein, C. 1981: The jackknife estimate of variance.Annals of Statistics 9, 586–596

    Google Scholar 

  18. Frank, P.D. 1995: Global modeling for optimization.SIAG/OPT Views-and-News 7, 9–12

    Google Scholar 

  19. Geisser, S.; Eddy, W.F. 1979: A predictive approach to model selection.J. Amer. Statistical Assoc. 74, 153–160

    Google Scholar 

  20. Giunta, A.A. 1997:Aircraft multidisciplinary optimization using design of experiments theory and response surface modeling methods. Ph.D. Thesis, Virginia Tech. Available as MAD 97-05-01, May 1997, Department of Aerospace and Ocean Engineering, Virginia Tech, 215 Randolph Hall, Blacksburg, Virginia 24061

    Google Scholar 

  21. Jones, D.R.; Schonlau, M.; Welch, W.J. 1997: A data analytic approach to Bayesian global optimization.Proc. ASA

  22. Koehler, J.R.; Owen, A.B. 1996: Computer experiments. In: Ghosh, S.; Rao, C.R. (eds.)Handbook of statistics, volume 13, pp. 261–308. New York: Elsevier Science

    Google Scholar 

  23. Levine, D. 1996: Users guide to the PGAPack parallel genetic algorithm library.Technical Report ANL-95/18, Argonne National Laboratory, 9700 South Cass Avenue, Argonne, Illinois 60439 Available from URL ftp: //info.mcs.anl.gov/pub/pgapack/pgapack.tar.Z

    Google Scholar 

  24. Lewis, R.M.; Torczon, V. 1996: Pattern search algorithms for bound constrained minimization.Technical Report 96-20 ICASE, NASA Langley Research Center, Hampton, Virginia 23681-2199. To appear inSIAM J. Optimiz.

    Google Scholar 

  25. Lewis, R.M.; Torczon, V. 1996: Rank ordering and positive bases in pattern search algorithms.Technical Report 96-71, ICASE, NASA Langley Research Center, Hampton, Virginia 23681-2199. In revision forMath. Prog.

    Google Scholar 

  26. Lewis, R.M.; Torczon, V. 1997: Pattern search methods for linearly constrained minimization.Technical Report 98-03, ICASE, NASA Langley Research Center, Hampton, Virginia 23681-2199.SIAM J. Optimiz. (to appear)

    Google Scholar 

  27. McKay, M.D.; Conover, W.J.; Beckman, R.J. 1979 A comparison of three methods for selecting values of input variables in the analysis of output from a computer code.Technometrics 21, 239–245

    Google Scholar 

  28. Owen, A.B. 1992: Orthogonal arrays for computer experiments, integration and visualization.Statistica Sinica 2, 439–452

    Google Scholar 

  29. Rinnooy Kan, A.H.G.; Timmer, G.T. 1984: A stochastic approach to global optimization. In: Boggs, P.T.; Byrd, R.H.; Schnabel, R.B. (eds.)Numerical optimization 1984 (Proc. SIAM Conf. on Numerical Optimization), pp. 245–262. Philadelphia: SIAM

    Google Scholar 

  30. Ripley, B.D. 1988:Statistical inference for spatial processes. Cambridge: Cambridge University Press

    Google Scholar 

  31. Sacks, J.; Welch, W.J.; Mitchell, T.J.; Wynn, H.P. 1989: Design and analysis of computer experiments.Statistical Sci.4, 409–435

    Google Scholar 

  32. Serafini, D.B. 1998:A framework for managing models in nonlinear optimization of computationally expensive functions. Ph.D. Thesis, Rice University

  33. Shultz, L.A.; Panda, B.; Tarzanin, F.J.; Derham, R.C.; Oh, B.K.; Dadone, L. 1994: Interdisciplinary analysis for advanced rotors-approach, capabilities and status. AHS Aeromechanics Specialists Conf., PS 4-1-4-15

  34. Snir, M.; Otto S.W.; Huss-Lederman, S.; Walker, D.W.; Dongarra, J. 1996.MPI: The complete reference. Cambridge, MA: The MIT Press

    Google Scholar 

  35. Stein, M. 1987: Large sample properties of simulations using latin hypercube sampling.Technometrics 29, 143–151

    Google Scholar 

  36. Tang, B. 1993: Orthogonal array-based latin hypercubes.J. Amer. Statistical Assoc. 88, 1392–1397

    Google Scholar 

  37. Torczon, V. 1992: PDS: direct search methods for unconstrained optimization on either sequential or parallel machines.Technical Report 92-9, Department of Computational and Applied Mathematics, Rice University, Houston, Texas 77005-1892. In revision forACM Trans. Math. Software

    Google Scholar 

  38. Torczon, V. 1995: Pattern search methods for nonlinear optimization.SIAG/OPT Views & News 6, 7–11

    Google Scholar 

  39. Torczon, V. 1997: On the convergence of pattern search algorithms.SIAM J. Optimiz. 7, 1–25

    Google Scholar 

  40. Torczon, V.; Trosset, M.W. 1998: From evolutionary operation to parallel direct search: pattern search algorithms for numerical optimization.Computing Sci. & Statistics 29, 396–401

    Google Scholar 

  41. Trosset, M.W.; Torczon, V. 1997: Numerical optimization using computer experiments.Technical Report 97-38, ICASE, NASA Langley Research Center, Hampton, Virginia 23681-2199

    Google Scholar 

  42. Trosset, M.W. 1997: I know it when I see it: toward a definition of direct search methods.SIAG/OPT Views & News 9, 7–10

    Google Scholar 

  43. Watson, G.S. 1984: Smoothing and interpolation by kriging and with splines.Math. Geology 16, 601–615

    Google Scholar 

Download references

Author information

Affiliations

Authors

Additional information

Communicated by J. Sobieski

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Booker, A.J., Dennis, J.E., Frank, P.D. et al. A rigorous framework for optimization of expensive functions by surrogates. Structural Optimization 17, 1–13 (1999). https://doi.org/10.1007/BF01197708

Download citation

Keywords

  • Objective Function
  • Design Problem
  • Optimization Approach
  • Engineering Design
  • Rotor Blade