Structural and Multidisciplinary Optimization

, Volume 45, Issue 1, pp 53–64 | Cite as

An efficient class of direct search surrogate methods for solving expensive optimization problems with CPU-time-related functions

  • Mark A. Abramson
  • Thomas J. Asaki
  • John E. DennisJr.
  • Raymond MagallanezJr.
  • Matthew J. Sottile
Research Paper


In this paper, we characterize a new class of computationally expensive optimization problems and introduce an approach for solving them. In this class of problems, objective function values may be directly related to the computational time required to obtain them, so that, as the optimal solution is approached, the computational time required to evaluate the objective is significantly less than at points farther away from the solution. This is motivated by an application in which each objective function evaluation requires both a numerical fluid dynamics simulation and an image registration process, and the goal is to find the parameter values of a predetermined reference image by comparing the flow dynamics from the numerical simulation and the reference image through the image comparison process. In designing an approach to numerically solve the more general class of problems in an efficient way, we make use of surrogates based on CPU times of previously evaluated points, rather than their function values, all within the search step framework of mesh adaptive direct search algorithms. Because of the expected positive correlation between function values and their CPU times, a time cutoff parameter is added to the objective function evaluation to allow its termination during the comparison process if the computational time exceeds a specified threshold. The approach was tested using the NOMADm and DACE MATLAB® software packages, and results are presented.


Surrogate optimization Derivative-free optimization Black box optimization Mesh Adaptive Direct Search (MADS) Pattern search Image registration Kriging 



The authors wish to thank David Bethea and two anonymous referees for some useful comments and discussions. Support for the first author was provided by Los Alamos National Laboratory (LANL). Support for the third author was provided by LANL, Air Force Office of Scientific Research F49620-01-1-0013, The Boeing Company, and ExxonMobil Upstream Research Company.

The views expressed in this document are those of the authors and do not reflect the official policy or position of the United States Air Force, Department of Defense, United States Government, or corporate affiliations of the authors.


  1. Abramson MA (2005) Second-order behavior of pattern search. SIAM J Optim 16(2):315–330CrossRefMathSciNetGoogle Scholar
  2. Abramson MA (2008) NOMADm optimization software.
  3. Abramson MA, Audet C (2006) Second-order convergence of mesh adaptive direct search. SIAM J Optim 17(2):606–619CrossRefzbMATHMathSciNetGoogle Scholar
  4. Abramson MA, Audet C, Dennis JE Jr, Le Digabel S (2009) OrthoMADS: a deterministic instance with orthogonal directions. SIAM J Optim 20(2):948–966CrossRefzbMATHMathSciNetGoogle Scholar
  5. Alexandrov N, Dennis JE Jr, Lewis R, Torczon V (1998) A trust region framework for managing the use of approximation models in optimization. Struct Optim 15:16–23CrossRefGoogle Scholar
  6. Audet C, Dennis JE Jr (2003) Analysis of generalized pattern searches. SIAM J Optim 13(3):889–903CrossRefzbMATHMathSciNetGoogle Scholar
  7. Audet C, Dennis JE Jr (2004) A pattern search filter method for nonlinear programming without derivatives. SIAM J Optim 14(4):980–1010CrossRefzbMATHMathSciNetGoogle Scholar
  8. Audet C, Dennis JE Jr (2006) Mesh adaptive direct search algorithms for constrained optimization. SIAM J Optim 17(2):188–217CrossRefzbMATHMathSciNetGoogle Scholar
  9. Bethea D (2008) Improving mixed variable optimization of computational and model parameters using multiple surrogate functions. Master’s thesis, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, OHGoogle Scholar
  10. Booker AJ (2000) Well-conditioned Kriging models for optimization of computer simulations. Technical Report M&CT-TECH-00-002, Boeing Computer Services, Research and Technology, M/S 7L–68, Seattle, Washington 98124Google Scholar
  11. Booker AJ, Dennis JE Jr, Frank PD, Moore DW, Serafini DB (1998) Managing surrogate objectives to optimize a helicopter rotor design—further experiments. AIAA Paper 1998–4717, Presented at the 8th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, St. LouisGoogle Scholar
  12. Booker AJ, Dennis JE Jr, Frank PD, Serafini DB, Torczon V, Trosset MW (1999) A rigorous framework for optimization of expensive functions by surrogates. Struct Optim 17(1):1–13CrossRefGoogle Scholar
  13. Coope ID, Price CJ (2000) Frame-based methods for unconstrained optimization. J Optim Theory Appl 107(2):261–274CrossRefzbMATHMathSciNetGoogle Scholar
  14. Fischer B, Modersitzki J (2003) Curvature based image registration. J Math Imaging Vis 18(1):81–85CrossRefzbMATHMathSciNetGoogle Scholar
  15. Forrester AIJ, Keane AJ (2009) Recent advances in surrogate-based optimization in aerospace sciences. Prog Aerosp Sci 45(1–3):50–79CrossRefGoogle Scholar
  16. Ginsbourger D, Helbert C, Carraro L (2008) Discrete mixtures of kernels for kriging-based optimization. Qual Reliab Eng Int 24(6):681–691CrossRefGoogle Scholar
  17. Griebel M, Dornseifer T, Neunhoeffer T (1998) Numerical Simulation in fluid dynamics: a practical introduction. SIAM, New YorkCrossRefGoogle Scholar
  18. Hager WW (1984) Condition estimates. SIAM J Sci Statist Comput 5(2):311–316CrossRefzbMATHMathSciNetGoogle Scholar
  19. Higham NJ, Tisseur F (2000) A block algorithm for matrix 1-norm estimation with an application to 1-norm pseudospectra. SIAM J Matrix Anal Appl 21(4):1185–1201CrossRefzbMATHMathSciNetGoogle Scholar
  20. Kleijnen JPC (2009) Kriging metamodeling in simulation: a review. Eur J Oper Res 192(3):707–716CrossRefzbMATHMathSciNetGoogle Scholar
  21. Lewis RM, Torczon V (1999) Pattern search algorithms for bound constrained minimization. SIAM J Optim 9(4):1082–1099CrossRefzbMATHMathSciNetGoogle Scholar
  22. Lewis RM, Torczon V (2000) Pattern search methods for linearly constrained minimization. SIAM J Optim 10(3):917–941CrossRefzbMATHMathSciNetGoogle Scholar
  23. Lewis RM, Torczon V (2002) A globally convergent augmented Lagrangian pattern search algorithm for optimization with general constraints and simple bounds. SIAM J Optim 12(4):1075–1089CrossRefzbMATHMathSciNetGoogle Scholar
  24. Lophaven SN, Nielsen HB, Søndergaard J (2002) Aspects of the MATLAB toolbox DACE. Technical Report IMM-TR-2002-13, Technical University of Denmark, CopenhagenGoogle Scholar
  25. Magallanez R (2007) Surrogate strategies for computationally expensive optimization problems with CPU-time correlated functions. Master’s thesis, Graduate School of Engineering and Management, Air Force Institute of TechnologyGoogle Scholar
  26. Marin VE, Rincon JA, Romero DA (2009) A comparison of metamodel-assisted prescreening criteria for multi-objective genetic algorithms. In: ASME 2009 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, San Diego, CA, 30 Aug–2 Sep 2009. DETC2009-87736Google Scholar
  27. Martin JD (2010) Robust kriging models. In: 51th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Orlando, FL, 12–15 April 2010. AIAA 2010-2854Google Scholar
  28. Martin JD, Simpson TW (2005) Use of kriging models to approximate deterministic computer models. AIAA J 43(4):853–863CrossRefGoogle Scholar
  29. Modersitzki J (2004) Numerical methods for image registration. Oxford University PressGoogle Scholar
  30. Nielsen HB (2007) DACE surrogate models.
  31. Owen AB (1992) Orthogonal arrays for computer experiments, integration, and visualization. Statist Sin 2:439–452zbMATHMathSciNetGoogle Scholar
  32. Robinson TD, Eldred MS, Willcox KE, Haimes R (2006) Strategies for multifidelity optimization with variable dimernsional hierarchical models. In: Proc 47th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conf. (2nd AIAA Multidisciplinary Design Optimization Specialist Conf.). Newport, Rhode IslandGoogle Scholar
  33. Romero DA, Amon CH, Finger S (2008) A study of covariance functions for multi-response metamodeling for simulation-based design and optimization. In: ASME 2008 International Design Engineering Technical Conference, Brooklyn, NY, 3–6 August 2008. DETC2008-50061.Google Scholar
  34. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4(4):409–435CrossRefzbMATHMathSciNetGoogle Scholar
  35. Santner TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. SpringerGoogle Scholar
  36. Schmit LA Jr, Miura H (1976) Approximation concepts for efficient structural synthesis. Technical Report CR-2552, NASAGoogle Scholar
  37. Stein M (1987) Large sample properties of simulations using Latin hypercube sampling. Technometrics 29(2):143–151CrossRefzbMATHMathSciNetGoogle Scholar
  38. Stein ML (1999) Interpolation of spatial data: some theory for kriging. SpringerGoogle Scholar
  39. Tang B (1993) Orthogonal array-based Latin hypercubes. J Am Stat Assoc 88(424):1392–1397CrossRefzbMATHGoogle Scholar
  40. Torczon V (1997) On the convergence of pattern search algorithms. SIAM J Optim 7(1):1–25CrossRefzbMATHMathSciNetGoogle Scholar
  41. Viana FAC, Haftka RT (2008) Using multiple surrogates for metamodeling. In: 7th ASMO-UK/ISSMO International Conference on Engineering Design Optimization, Bath, UK, 7–8 July 2008Google Scholar
  42. Viana FAC, Gogu C, Haftka RT (2010a) Making the most out of surrogate models: tricks of the trade. In: ASME 2010 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, Montreal, Canada, 16–18 August 2010. DETC2010-28813Google Scholar
  43. Viana FAC, Haftka RT, Watson LT (2010b) Why not run the efficient global optimization algorithm with multiple surrogates? In: 51th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Orlando, FL, 12–15 April 2010. AIAA 2010-3090Google Scholar
  44. Voutchkov I, Keane AJ (2006) Multiobjective optimization using surrogates. In: 7th international conference on adaptive computing in design and manufacture, Bristol, UK, pp 167–175Google Scholar
  45. Wang GG, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129(4):370–380CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag (outside the USA) 2011

Authors and Affiliations

  • Mark A. Abramson
    • 1
  • Thomas J. Asaki
    • 2
  • John E. DennisJr.
    • 3
  • Raymond MagallanezJr.
    • 4
  • Matthew J. Sottile
    • 5
  1. 1.The Boeing CompanySeattleUSA
  2. 2.Department of MathematicsWashington State UniversityPullmanUSA
  3. 3.Department of Computational and Applied MathematicsRice UniversitySeattleUSA
  4. 4.Department of Mathematical SciencesUnited States Air Force AcademyColorado SpringsUSA
  5. 5.Galois, Inc.PortlandUSA

Personalised recommendations