A comparison of quality measures for model selection in surrogate-assisted evolutionary algorithm

Abstract

Choosing a proper approximation model should be the first and the most fundamental problem to be solved when dealing with surrogate-assisted evolutionary algorithms. Till now, most of the model selection methods emphasize on obtaining the best surrogate model basing on model accuracy assessments. As the population ranking is of the most important part in evolutionary optimization, the target function of surrogate model should focus on the right ranking of candidate solutions. Therefore, in this paper, we make a comparison study on several model quality measures which basically dedicated to measuring the capability of surrogate model in selecting and ranking the candidate solutions. In order to investigate the compatibility between accuracy assessments and ranking correlation methods, four algorithms with different model selection strategies based on different quality measures are designed and comparative study is made by contrasting them to three specific surrogate-assisted evolutionary algorithms as well as the standard particle swarm optimization. Simulation results on ten commonly used benchmark problems and one engineering case demonstrate the efficacy of the designed model selection strategies and meanwhile provide further insight into the three model quality measures studied in this paper in model selection.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. Buche D, Schraudolph NN, Koumoutsakos P (2005) Accelerating evolutionary algorithms with Gaussian process fitness function models. IEEE Trans Syst Man Cybern C 35:183–194. https://doi.org/10.1109/TSMCC.2004.841917

    Article  Google Scholar 

  2. Clerc M (1999) The swarm and the queen: towards a deterministic and adaptive particle swarm optimization. In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), vol 1953, pp 1–1957. https://doi.org/10.1109/cec.1999.785513

  3. Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol Comput 1:3–18. https://doi.org/10.1016/j.swevo.2011.02.002

    Article  Google Scholar 

  4. Díaz-Manríquez A, Toscano-Pulido G, Gómez-Flores W (2011) On the selection of surrogate models in evolutionary optimization algorithms. In: 2011 IEEE congress of evolutionary computation (CEC), pp 2155–2162. https://doi.org/10.1109/cec.2011.5949881

  5. Díaz-Manríquez A, Toscano G, Coello Coello CA (2016) Comparison of metamodeling techniques in evolutionary algorithms. Soft Comput. https://doi.org/10.1007/s00500-016-2140-z

    Article  Google Scholar 

  6. Díaz-Manríquez A, Toscano G, Coello Coello CA (2017) Comparison of metamodeling techniques in evolutionary algorithms. Soft Comput 21:5647–5663. https://doi.org/10.1007/s00500-016-2140-z

    Article  Google Scholar 

  7. Efron B, Tibshirani RJ (1994) An introduction to the bootstrap. CRC Press, Boca Raton

    Google Scholar 

  8. Emmerich MTM, Giannakoglou KC, Naujoks B (2006) Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels. IEEE Trans Evol Comput 10:421–439. https://doi.org/10.1109/TEVC.2005.859463

    Article  Google Scholar 

  9. Fisher NI, Hall P (1991) Bootstrap algorithms for small samples. J Stat Plan Inference 27:157–169. https://doi.org/10.1016/0378-3758(91)90013-5

    MathSciNet  Article  Google Scholar 

  10. Forrester AIJ, Keane AJ (2009) Recent advances in surrogate-based optimization. Prog Aerosp Sci 45:50–79. https://doi.org/10.1016/j.paerosci.2008.11.001

    Article  Google Scholar 

  11. Hüsken M, Jin Y, Sendhoff B (2005) Structure optimization of neural networks for evolutionary design optimization. Soft Comput 9:21–28. https://doi.org/10.1007/s00500-003-0330-y

    Article  Google Scholar 

  12. Ingu T, Takagi H (1999) Accelerating a GA convergence by fitting a single-peak function. In: Fuzzy systems conference proceedings, FUZZ-IEEE ‘99. 1999 IEEE International, vol 1413, pp 1415–1420. https://doi.org/10.1109/fuzzy.1999.790111

  13. Jin Y (2005) A comprehensive survey of fitness approximation in evolutionary computation. Soft Comput 9:3–12

    Article  Google Scholar 

  14. Jin Y (2011) Surrogate-assisted evolutionary computation: recent advances and future challenges. Swarm Evol Comput 1:61–70. https://doi.org/10.1016/j.swevo.2011.05.001

    Article  Google Scholar 

  15. Jin Y, Michael H (2003) Quality measures for approximate models in evolutionary computation. In: GECCO, pp 170–173

  16. Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodelling techniques under multiple modelling criteria. Struct Multidiscipl Optim 23:1–13. https://doi.org/10.1007/s00158-001-0160-4

    Article  Google Scholar 

  17. Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Global Optim 13:455–492. https://doi.org/10.1023/a:1008306431147

    MathSciNet  Article  MATH  Google Scholar 

  18. Lawrence CT, Tits AL (2001) A computationally efficient feasible sequential quadratic programming algorithm. SIAM J Optim 11:1092–1118. https://doi.org/10.1137/s1052623498344562

    MathSciNet  Article  MATH  Google Scholar 

  19. Le MN, Ong YS, Menzel S, Jin Y, Sendhoff B (2013) Evolution by adapting surrogates. Evol Comput 21:313–340. https://doi.org/10.1162/EVCO_a_00079

    Article  Google Scholar 

  20. Lendasse A, Wertz V, Verleysen M (2003) Model selection with cross-validations and bootstraps: application to time series prediction with RBFN models. In: Artificial neural networks and neural information processing–ICANN/ICONIP 2003, pp 573–580. Springer, Berlin

  21. Lesh FH (1959) Multi-dimensional least-squares polynomial curve fitting. Commun ACM 2:29–30. https://doi.org/10.1145/368424.368443

    Article  MATH  Google Scholar 

  22. Liang K-H, Yao X, Newton C (2000) Evolutionary search of approximated n-dimensional landscapes. Int J Knowl Based Intell Eng Syst 4:172–183

    Google Scholar 

  23. Lim D, Ong Y-S, Jin Y, Sendhoff B (2007) A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation. Paper presented at the Proceedings of the 9th annual conference on Genetic and evolutionary computation, London, England

  24. Lim D, Ong Y-S, Jin Y, Sendhoff B (2008) Evolutionary optimization with dynamic fidelity computational models. In: Huang D-S, Wunsch DC, Levine DS, Jo K-H (eds) Advanced intelligent computing theories and applications. With Aspects of artificial intelligence: 4th international conference on intelligent computing, ICIC 2008 Shanghai, China, September 15-18, 2008. Springer, Berlin, pp 235–242 https://doi.org/10.1007/978-3-540-85984-0_29

  25. Lim D, Jin Y, Ong YS, Sendhoff B (2010) Generalizing surrogate-assisted evolutionary computation. IEEE Trans Evol Comput 14:329–355. https://doi.org/10.1109/TEVC.2009.2027359

    Article  Google Scholar 

  26. Liu H, Hervas J-R, Ong Y-S, Cai J, Wang Y (2018) An adaptive RBF-HDMR modeling approach under limited computational budget. Struct Multidiscipl Optim 57:1233–1250. https://doi.org/10.1007/s00158-017-1807-0

    Article  Google Scholar 

  27. Lophaven SN, Nielsen HB, Søndergaard J (2002) DACE-A Matlab Kriging toolbox, version 2.0

  28. Matheron G (1963) Principles of geostatistics. Econ Geol 58:1246–1266

    Article  Google Scholar 

  29. Mokarram V, Banan MR (2018) A new PSO-based algorithm for multi-objective optimization with continuous and discrete design variables. Struct Multidiscipl Optim 57:509–533. https://doi.org/10.1007/s00158-017-1764-7

    MathSciNet  Article  Google Scholar 

  30. Munk DJ, Vio GA, Steven GP (2015) Topology and shape optimization methods using evolutionary algorithms: a review. Struct Multidiscipl Optim 52:613–631. https://doi.org/10.1007/s00158-015-1261-9

    MathSciNet  Article  Google Scholar 

  31. Nakayama H, Arakawa M, Sasaki R (2001) A computational intelligence approach to optimization with unknown objective functions. In: Artificial neural networks—ICANN 2001. Springer, Berlin, pp 73–80

  32. Powell M (2001) Radial basis function methods for interpolation to functions of many variables. In: HERCMA. Citeseer, pp 2–24

  33. Praveen C, Duvigneau R (2009) Low cost PSO using metamodels and inexact pre-evaluation: application to aerodynamic shape design. Comput Methods Appl Mech Eng 198:1087–1096. https://doi.org/10.1016/j.cma.2008.11.019

    Article  MATH  Google Scholar 

  34. Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Kevin Tucker P (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41:1–28. https://doi.org/10.1016/j.paerosci.2005.02.001

    Article  MATH  Google Scholar 

  35. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4:409–423

    MathSciNet  Article  Google Scholar 

  36. Sadoughi M, Hu C, MacKenzie CA, Eshghi AT, Lee S (2018) Sequential exploration-exploitation with dynamic trade-off for efficient reliability analysis of complex engineered systems. Struct Multidiscipl Optim 57:235–250. https://doi.org/10.1007/s00158-017-1748-7

    MathSciNet  Article  Google Scholar 

  37. Shi L, Rasheed K (2008) ASAGA: an adaptive surrogate-assisted genetic algorithm. Paper presented at the Proceedings of the 10th annual conference on Genetic and evolutionary computation, Atlanta, GA, USA

  38. Shi L, Rasheed K (2010) A survey of fitness approximation methods applied in evolutionary algorithms. In: Tenne Y, Goh C-K (eds) Computational intelligence in expensive optimization problems. Springer, Berlin, pp 3–28. https://doi.org/10.1007/978-3-642-10701-6_1

  39. Simpson TW, Poplinski JD, Koch PN, Allen JK (2001) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17:129–150. https://doi.org/10.1007/pl00007198

    Article  MATH  Google Scholar 

  40. Stern RE, Song J, Work DB (2017) Accelerated Monte Carlo system reliability analysis through machine-learning-based surrogate models of network connectivity. Reliab Eng Syst Safe 164:1–9. https://doi.org/10.1016/j.ress.2017.01.021

    Article  Google Scholar 

  41. Tang Y, Chen J, Wei J (2013) A surrogate-based particle swarm optimization algorithm for solving optimization problems with expensive black box functions. Eng Optim 45:557–576. https://doi.org/10.1080/0305215X.2012.690759

    MathSciNet  Article  Google Scholar 

  42. Wang GG, Shan S (2006) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129:370–380. https://doi.org/10.1115/1.2429697

    Article  Google Scholar 

  43. Wang D, Wu Z, Fei Y, Zhang W (2014) Structural design employing a sequential approximation optimization approach. Comput Struct 134:75–87. https://doi.org/10.1016/j.compstruc.2013.12.004

    Article  Google Scholar 

  44. Wang H, Jin Y, Doherty J (2017) Committee-based active learning for surrogate-assisted particle swarm optimization of expensive problems. IEEE Trans Cybern 47:2664–2677. https://doi.org/10.1109/TCYB.2017.2710978

    Article  Google Scholar 

  45. Wang H, Jin Y, Sun C, Doherty J (2018) Offline data-driven evolutionary optimization using selective surrogate ensembles. IEEE Trans Evol Comput. https://doi.org/10.1109/tevc.2018.2834881

    Article  Google Scholar 

  46. Wu Z, Wang D, Okolo NP, Hu F, Zhang W (2016) Global sensitivity analysis using a gaussian radial basis function metamodel. Reliab Eng Syst Safe 154:171–179. https://doi.org/10.1016/j.ress.2016.06.006

    Article  Google Scholar 

  47. Yang D, Flockton SJ (1995) Evolutionary algorithms with a coarse-to-fine function smoothing. In: IEEE international conference on evolutionary computation. IEEE, pp 657–662

  48. Yang Q, Xue D (2015) Comparative study on influencing factors in adaptive metamodeling. Eng Comput 31:561–577. https://doi.org/10.1007/s00366-014-0358-x

    Article  Google Scholar 

  49. Yang Q, Kianimanesh A, Freiheit T, Park SS, Xue D (2011) A semi-empirical model considering the influence of operating parameters on performance for a direct methanol fuel cell. J Power Sources 196:10640–10651. https://doi.org/10.1016/j.jpowsour.2011.08.104

    Article  Google Scholar 

  50. Yew Soon O, Keane AJ (2004) Meta-Lamarckian learning in memetic algorithms. IEEE Trans Evol Comput 8:99–110. https://doi.org/10.1109/TEVC.2003.819944

    Article  Google Scholar 

  51. Younis A, Dong Z (2010) Trends, features, and tests of common and recently introduced global optimization methods. Eng Optim 42:691–718. https://doi.org/10.1080/03052150903386674

    MathSciNet  Article  Google Scholar 

  52. Yu H, Tan Y, Sun C, Zeng J, Jin Y (2016) An adaptive model selection strategy for surrogate-assisted particle swarm optimization algorithm. In: 2016 IEEE symposium series on computational intelligence (SSCI), pp 1–8 https://doi.org/10.1109/ssci.2016.7850208

  53. Yu H, Tan Y, Sun C, Zeng J (2018a) A generation-based optimal restart strategy for surrogate-assisted social learning particle swarm optimization. Knowl Based Syst 10:1–12. https://doi.org/10.1016/j.knosys.2018.08.010

    Article  Google Scholar 

  54. Yu H, Tan Y, Zeng J, Sun C, Jin Y (2018b) Surrogate-assisted hierarchical particle swarm optimization. Inf Sci 454–455:59–72. https://doi.org/10.1016/j.ins.2018.04.062

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

This work is supported by National Natural Science Foundation of China (Grant Nos. 61472269 and 61403272) and the State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, China, as well as the Fund Program for the Scientific Activities of Selected Returned Overseas Professionals in Shanxi Province.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Haibo Yu.

Ethics declarations

Conflict of interest

Author Haibo Yu declares that he has no conflict of interest. Author Ying Tan declares that she has no conflict of interest. Author Chaoli Sun declares that she has no conflict of interest. Author Jianchao Zeng declares that he has no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by V. Loia.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yu, H., Tan, Y., Sun, C. et al. A comparison of quality measures for model selection in surrogate-assisted evolutionary algorithm. Soft Comput 23, 12417–12436 (2019). https://doi.org/10.1007/s00500-019-03783-0

Download citation

Keywords

  • Surrogate
  • Model selection
  • Bootstrap sampling
  • Quality measure
  • Particle swarm optimization