Journal of Global Optimization

, Volume 70, Issue 3, pp 645–675 | Cite as

Order-based error for managing ensembles of surrogates in mesh adaptive direct search

  • Charles Audet
  • Michael Kokkolaras
  • Sébastien Le Digabel
  • Bastien Talgorn


We investigate surrogate-assisted strategies for global derivative-free optimization using the mesh adaptive direct search (MADS) blackbox optimization algorithm. In particular, we build an ensemble of surrogate models to be used within the search step of MADS to perform global exploration, and examine different methods for selecting the best model for a given problem at hand. To do so, we introduce an order-based error tailored to surrogate-based search. We report computational experiments for ten analytical benchmark problems and three engineering design applications. Results demonstrate that different metrics may result in different model choices and that the use of order-based metrics improves performance.


Derivate-free optimization Ensemble of surrogates MADS Order error 



This work has been supported partially by FRQNT Grant 2015-PR-182098; B. Talgorn and M. Kokkolaras are grateful for the partial support of NSERC/Hydro-Québec Grant EGP2 498903-16; such support does not constitute an endorsement by the sponsors of the opinions expressed in this article. The authors would like to thank Robin Tournemenne (École Centrale de Nantes) for his insightful questions and comments that contributed significantly to improve the search algorithm used in this work. The authors would also like to express their gratitude to Stéphane Alarie (IREQ) for his support, which made this work possible.

Supplementary material


  1. 1.
    Abramson, M.A., Audet, C., Dennis Jr., J.E., Le Digabel, S.: OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM J. Optim. 20(2), 948–966 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Acar, E., Rais-Rohani, M.: Ensemble of metamodels with optimized weight factors. Struct. Multidiscipl. Optim. 37(3), 279–294 (2009)CrossRefGoogle Scholar
  3. 3.
    Agte, J.S., Sobieszczanski-Sobieski, J., Sandusky, R.R.J.: Supersonic business jet design through bilevel integrated system synthesis. In: Proceedings of the World Aviation Conference, Volume SAE Paper No. 1999-01-5622, San Francisco, CA, 1999. MCB University Press, BradfordGoogle Scholar
  4. 4.
    Alexandrov, N.M., Dennis Jr., J.E., Lewis, R.M., Torczon, V.: A trust-region framework for managing the use of approximation models in optimization. Struct. Multidiscipl. Optim. 15(1), 16–23 (1998)CrossRefGoogle Scholar
  5. 5.
    Allen, D.M.: The relationship between variable selection and data agumentation and a method for prediction. Technometrics 16(1), 125–127 (1974)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Audet, C.: A survey on direct search methods for blackbox optimization and their applications. In: Pardalos, P.M., Rassias, T.M. (eds.) Mathematics Without Boundaries: Surveys in Interdisciplinary Research, chapter 2, pp. 31–56. Springer, Berlin (2014)Google Scholar
  7. 7.
    Audet, C., Béchard, V., Le Digabel, S.: Nonsmooth optimization through mesh adaptive direct search and variable neighborhood search. J. Glob. Optim. 41(2), 299–318 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Audet, C., Booker, A.J., Dennis Jr., J.E., Frank, P.D., Moore, D.W.: A surrogate-model-based method for constrained optimization. Presented at the 8th AIAA/ISSMO symposium on multidisciplinary analysis and optimization, 2000Google Scholar
  9. 9.
    Audet, C., Dennis Jr., J.E.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Audet, C., Dennis Jr., J.E.: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20(1), 445–472 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Bauer, F., Garabedian, P., Korn, D.: Supercritical Wing Section, vol. III. Springer, Berlin (1977)CrossRefzbMATHGoogle Scholar
  12. 12.
    Booker, A.J., Dennis Jr., J.E., Frank, P.D., Serafini, D.B., Torczon, V., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Multidiscipl. Optim. 17(1), 1–13 (1999)CrossRefGoogle Scholar
  13. 13.
    Boukouvalaa, F., Floudas, C.A.: ARGONAUT: AlgoRithms for Global Optimization of coNstrAined grey-box compUTational problems. Optim. Lett. 11(5), 895–913 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Boukouvalaa, F., Hasan, M.M.F., Floudas, C.A.: Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption. J. Glob. Optim. 67(1), 3–42 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Buhmann, M.D.: Radial Basis Functions: Theory and Implementations. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge (2003)CrossRefGoogle Scholar
  16. 16.
    Chen, Y.-C., Wei, C., Yeh, H.-C.: Rainfall network design using kriging and entropy. Hydrol. Process. 22(3), 340–346 (2008)CrossRefGoogle Scholar
  17. 17.
    Chowdhury, S., Mehmani, A., Zhang, J., Messac, A.: Quantifying regional error in surrogates by modeling its relationship with sample density. In: Structures, Structural Dynamics, and Materials and Co-located Conferences. American Institute of Aeronautics and Astronautics (2013)Google Scholar
  18. 18.
    Colson, B.: Trust-region algorithms for derivative-free optimization and nonlinear bilevel programming. Ph.D. thesis, Département de Mathématique, FUNDP, Namur, Belgium (2003)Google Scholar
  19. 19.
    Conn, A.R., Le Digabel, S.: Use of quadratic models with mesh-adaptive direct search for constrained black box optimization. Optim. Methods Softw. 28(1), 139–158 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MOS-SIAM Series on Optimization. SIAM, Philadelphia (2009)CrossRefzbMATHGoogle Scholar
  21. 21.
    Conn, A.R., Toint, P.L.: An algorithm using quadratic interpolation for unconstrained derivative free optimization. In: Di Pillo, G., Gianessi, F. (eds.) Nonlinear Optimization and Applications, pp. 27–47. Plenum Publishing, New York (1996)CrossRefGoogle Scholar
  22. 22.
    Custódio, A.L., Rocha, H., Vicente, L.N.: Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. 46(2), 265–278 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Dhar, A., Datta, B.: Global optimal design of ground water monitoring network using embedded kriging. Ground Water 47(6), 806–815 (2009)CrossRefGoogle Scholar
  24. 24.
    Eaves, B.C.: On quadratic programming. Manag. Sci. 17(11), 698–711 (1971)CrossRefzbMATHGoogle Scholar
  25. 25.
    Efron, B.: Estimating the error rate of a prediction rule: improvement on cross-validation. J. Am. Stat. Assoc. 78(382), 316–331 (1983)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Fletcher, R., Gould, N.I.M., Leyffer, S., Toint, P.L., Wächter, A.: On the global convergence of trust-region SQP-filter algorithms for general nonlinear programming. SIAM J. Optim. 13(3), 635–659 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Fowler, K.R., Reese, J.P., Kees, C.E., Dennis Jr., J.E., Kelley, C.T., Miller, C.T., Audet, C., Booker, A.J., Couture, G., Darwin, R.W., Farthing, M.W., Finkel, D.E., Gablonsky, J.M., Gray, G., Kolda, T.G.: Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv. Water Resour. 31(5), 743–757 (2008)CrossRefGoogle Scholar
  28. 28.
    Gergonne, J.D.: The application of the method of least squares to the interpolation of sequences. Hist. Math. 1(4), 439–447 (1974)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Goel, T., Haftka, R.T., Shyy, W., Queipo, N.V.: Ensemble of surrogates. Struct. Multidiscipl. Optim. 33(3), 199–216 (2007)CrossRefGoogle Scholar
  30. 30.
    Goel, T., Stander, N.: Comparing three error criteria for selecting radial basis function network topology. Comput. Methods Appl. Mech. Eng. 198(27–29), 2137–2150 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Goldberg, D.E.: Genetic Algorithms in Search. Optimization and Machine Learning. Addison-Wesley Longman, Boston (1989)zbMATHGoogle Scholar
  32. 32.
    Gramacy, R.B., Gray, G.A., Le Digabel, S., Lee, H.K.H., Ranjan, P., Wells, G., Wild, S.M.: Modeling an augmented lagrangian for blackbox constrained optimization. Technometrics 58(1), 1–11 (2016)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Gramacy, R.B., Le Digabel, S.: The mesh adaptive direct search algorithm with treed Gaussian process surrogates. Pac. J. Optim. 11(3), 419–447 (2015)MathSciNetzbMATHGoogle Scholar
  34. 34.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer Series in Statistics. Springer, New York (2001)zbMATHGoogle Scholar
  35. 35.
    Hock, W., Schittkowski, K.: Test Examples for Nonlinear Programming Codes. Lecture Notes in Economics and Mathematical Systems, vol. 187. Springer, Berlin (1981)CrossRefzbMATHGoogle Scholar
  36. 36.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black box functions. J. Glob. Optim. 13(4), 455–492 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Kannan, A., Wild, S.M.: Benefits of deeper analysis in simulation-based groundwater optimization problems. In: Proceedings of the XIX International Conference on Computational Methods in Water Resources (CMWR 2012), June 2012Google Scholar
  38. 38.
    Kitayama, S., Arakawa, M., Yamazaki, K.: Sequential approximate optimization using radial basis function network for engineering optimization. Optim. Eng. 12(4), 535–557 (2011)CrossRefzbMATHGoogle Scholar
  39. 39.
    Kodiyalam, S.: Multidisciplinary aerospace systems optimization. Technical Report NASA/CR-2001-211053, Lockheed Martin Space Systems Company, Computational AeroSciences Project, Sunnyvale, CA (2001)Google Scholar
  40. 40.
    Kroo, I.: Aircraft design: synthesis and analysis; Cruise performance and range. (2005)
  41. 41.
    Le Digabel, S.: Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37(4), 44:1–44:15 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Le Thi, H.A., Vaz, A.I.F., Vicente, L.N.: Optimizing radial basis functions by D.C. programming and its use in direct search for global derivative-free optimization. TOP 20(1), 190–214 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Lim, D., Ong, Y.-S., Jin, Y., Sendhoff, B.: A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, GECCO ’07, pp. 1288–1295. ACM, New York (2007)Google Scholar
  44. 44.
    Lukšan, L., Vlček, J.: Test problems for nonsmooth unconstrained and linearly constrained optimization. Technical Report V-798, ICS AS CR, 2000Google Scholar
  45. 45.
    Mack, Y., Goel, T., Shyy, W., Haftka, R.T., Queipo, N.V.: Multiple surrogates for the shape optimization of bluff body-facilitated mixing. In: 43rd AIAA Aerospace Sciences Meeting and Exhibit, Reno, Nevada, 2005. AIAA. Paper AIAA-2005-0333Google Scholar
  46. 46.
    Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate Analysis. Probability and mathematical statistics. Academic Press, New York (1979)zbMATHGoogle Scholar
  47. 47.
    Matott, L.S., Leung, K., Sim, J.: Application of MATLAB and Python optimizers to two case studies involving groundwater flow and contaminant transport modeling. Comput. Geosci. 37(11), 1894–1899 (2011)CrossRefGoogle Scholar
  48. 48.
    Matott, L.S., Rabideau, A.J., Craig, J.R.: Pump-and-treat optimization using analytic element method flow models. Adv. Water Resour. 29(5), 760–775 (2006)CrossRefGoogle Scholar
  49. 49.
    McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979)MathSciNetzbMATHGoogle Scholar
  50. 50.
    Mehmani, A., Chowdhury, S., Messac, A.: Predictive quantification of surrogate model fidelity based on modal variations with sample density. Struct. Multidiscipl. Optim. 52(2), 353–373 (2015)CrossRefGoogle Scholar
  51. 51.
    Mladenović, N., Hansen, P.: Variable neighborhood search. Comput. Oper. Res. 24(11), 1097–1100 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  52. 52.
    Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  53. 53.
    Müller, J., Piché, R.: Mixture surrogate models based on Dempster-Shafer theory for global optimization problems. J. Glob. Optim. 51(1), 79–104 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  54. 54.
    Müller, J., Shoemaker, C.A.: Influence of ensemble surrogate models and sampling strategy on the solution quality of algorithms for computationally expensive black-box global optimization problems. J. Glob. Optim. 60(2), 123–144 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  55. 55.
    Orr, M.J.L.: Introduction to radial basis function networks. Technical report, Center for Cognitive Science, University of Edinburgh (1996)Google Scholar
  56. 56.
    Peremezhney, N., Hines, E., Lapkin, A., Connaughton, C.: Combining gaussian processes, mutual information and a genetic algorithm for multi-target optimization of expensive-to-evaluate functions. Eng. Optim. 46(11), 1593–1607 (2014)MathSciNetCrossRefGoogle Scholar
  57. 57.
    Queipo, N.V., Haftka, R.T., Shyy, W., Goel, T., Vaidyanathan, R., Tucher, P.K.: Surrogate-based analysis and optimization. Prog. Aerosp. Sci. 41(1), 1–28 (2005)CrossRefGoogle Scholar
  58. 58.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press, Cambridge (2006)zbMATHGoogle Scholar
  59. 59.
    Regis, R.G.: Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions. Comput. Oper. Res. 38(5), 837–853 (2011)MathSciNetCrossRefGoogle Scholar
  60. 60.
    Regis, R.G.: Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points. Eng. Optim. 46(2), 218–243 (2014)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Shi, R., Liu, L., Long, T., Liu, J.: An efficient ensemble of radial basis functions method based on quadratic programming. Eng. Optim. 48(7), 1202–1225 (2016)MathSciNetCrossRefGoogle Scholar
  62. 62.
    Stigler, S.M.: Gergonne’s 1815 paper on the design and analysis of polynomial regression experiments. Hist. Math. 1(4), 431–439 (1974)MathSciNetCrossRefzbMATHGoogle Scholar
  63. 63.
    Stone, M.: An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. J. R. Stat. Soc. Ser. B (Methodol.) 39(1), 44–47 (1977)MathSciNetzbMATHGoogle Scholar
  64. 64.
    Taddy, M.A., Gramacy, R.B., Polson, N.G.: Dynamic trees for learning and design. J. Am. Stat. Assoc. 106(493), 109–123 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  65. 65.
    Talgorn, B., Le Digabel, S., Kokkolaras, M.: Statistical surrogate formulations for simulation-based design optimization. J. Mech. Des. 137(2), 021405-1–021405-18 (2015)CrossRefGoogle Scholar
  66. 66.
    Tarpey, T.: A note on the prediction sum of squares statistic for restricted least squares. Am. Stat. 54(2), 116–118 (2000)MathSciNetGoogle Scholar
  67. 67.
    Tenne, Y.: An adaptive-topology ensemble algorithm for engineering optimization problems. Optim. Eng. 16(2), 303–334 (2014)CrossRefzbMATHGoogle Scholar
  68. 68.
    Tosserams, S., Etman, L.F.P., Rooda, J.E.: A classification of methods for distributed system optimization based on formulation structure. Struct. Multidiscipl. Optim. 39(5), 503–517 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  69. 69.
    Tournemenne, R., Petiot, J.-F., Talgorn, B., Kokkolaras, M., Gilbert, J.: Brass instruments design using physics-based sound simulation models and surrogate-assisted derivative-free optimization. J. Mech. Des. 139(4), 041401-01–04140-19 (2017)CrossRefGoogle Scholar
  70. 70.
    Tribes, C., Dubé, J.-F., Trépanier, J.-Y.: Decomposition of multidisciplinary optimization problems: formulations and application to a simplified wing design. Eng. Optim. 37(8), 775–796 (2005)MathSciNetCrossRefGoogle Scholar
  71. 71.
    Vaz, A.I.F., Vicente, L.N.: A particle swarm pattern search method for bound constrained global optimization. J. Glob. Optim. 39(2), 197–219 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  72. 72.
    Viana, F.A.C., Haftka, R.T., Valder Jr., S., Butkewitsch, S., Leal, M.F.: Ensemble of Surrogates: a Framework based on Minimization of the Mean Integrated Square Error. In: 49th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials, Schaumburg, IL (2008)Google Scholar
  73. 73.
    Viana, F.A.C., Haftka, R.T., Watson, L.T.: Efficient global optimization algorithm assisted by multiple surrogate techniques. J. Glob. Optim. 56(2), 669–689 (2013)CrossRefzbMATHGoogle Scholar
  74. 74.
    Watson, G.S.: Smoothing and interpolation by kriging and with splines. Math. Geol. 16, 601–615 (1984)MathSciNetCrossRefGoogle Scholar
  75. 75.
    Wild, S.M., Regis, R.G., Shoemaker, C.A.: ORBIT: optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30(6), 3197–3219 (2008)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.GERAD and Département de Mathématiques et de Génie IndustrielÉcole Polytechnique de MontréalMontrealCanada
  2. 2.GERAD and Department of Mechanical EngineeringMcGill UniversityMontrealCanada

Personalised recommendations