Skip to main content
SpringerLink
Log in

On Statistical Analysis of MOEAs with Multiple Performance Indicators

  • Conference paper
  • First Online:
Evolutionary Multi-Criterion Optimization (EMO 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12654))

Included in the following conference series:

Abstract

Assessing the empirical performance of Multi-Objective Evolutionary Algorithms (MOEAs) is vital when we extensively test a set of MOEAs and aim to determine a proper ranking thereof. Multiple performance indicators, e.g., the generational distance and the hypervolume, are frequently applied when reporting the experimental data, where typically the data on each indicator is analyzed independently from other indicators. Such a treatment brings conceptual difficulties in aggregating the result on all performance indicators, and it might fail to discover significant differences among algorithms if the marginal distributions of the performance indicator overlap. Therefore, in this paper, we propose to conduct a multivariate \(\mathcal {E}\)-test on the joint empirical distribution of performance indicators to detect the potential difference in the data, followed by a post-hoc procedure that utilizes the linear discriminative analysis to determine the superiority between algorithms. This performance analysis’s effectiveness is supported by an experimentation conducted on four algorithms, 16 problems, and 6 different numbers of objectives.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The data files and the source code of this study can be accessed here: https://github.com/wangronin/EMO21.

  2. 2.

    For applying the DSC ranking scheme, we used the DSCTool, which is developed as RESTful web services to the DSC ranking functionalities [14].

References

  1. Beume, N., Naujoks, B., Emmerich, M.: SMS-EMOA: multiobjective selection based on dominated hypervolume. Eur. J. Oper. Res. 181(3), 1653–1669 (2007)

    Article  Google Scholar 

  2. Bezerra, L.C.T., López-Ibáñez, M., Stützle, T.: Automatic component-wise design of multiobjective evolutionary algorithms. IEEE Trans. Evol. Comput. 20(3), 403–417 (2016)

    Article  Google Scholar 

  3. Brans, J.-P., Mareschal, B.: Promethee methods. Multiple Criteria Decision Analysis: State of the Art Surveys. ISORMS, vol. 78, pp. 163–186. Springer, New York (2005). https://doi.org/10.1007/0-387-23081-5_5

    Chapter  Google Scholar 

  4. Coello, C.A.C., Cortés, N.C.: Solving multiobjective optimization problems using an artificial immune system. Genet. Program. Evolvable Mach. 6(2), 163–190 (2005)

    Article  Google Scholar 

  5. Coello, C.A.C., van Veldhuizen, D.A., Lamont, G.B.: Evolutionary algorithms for solving multi-objective problems, Genetic algorithms and evolutionary computation. Kluwer, vol. 5 (2002)

    Google Scholar 

  6. D’Agostino, R.B.: Goodness-of-Fit-Techniques, vol. 68. CRC Press, United States (1986)

    MATH  Google Scholar 

  7. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)

    Article  Google Scholar 

  8. Deb, K., Jain, H.: An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach. IEEE Trans. Evol. Comput. 18(4), 577–601 (2014)

    Article  Google Scholar 

  9. Deb, K., Sindhya, K., Hakanen, J.: Multi-objective optimization. In: Decision Sciences: Theory and Practice, pp. 145–184. CRC Press (2016)

    Google Scholar 

  10. Eftimov, T., Korošec, P., Seljak, B.K.: Comparing multi-objective optimization algorithms using an ensemble of quality indicators with deep statistical comparison approach. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8. IEEE (2017)

    Google Scholar 

  11. Eftimov, T., Korošec, P., Koroušić Seljak, B.: Deep statistical comparison applied on quality indicators to compare multi-objective stochastic optimization algorithms. In: Nicosia, G., Pardalos, P., Giuffrida, G., Umeton, R. (eds.) MOD 2017. LNCS, vol. 10710, pp. 76–87. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-72926-8_7

    Chapter  Google Scholar 

  12. Eftimov, T., Korošec, P., Seljak, B.K.: A novel approach to statistical comparison of meta-heuristic stochastic optimization algorithms using deep statistics. Inf. Sci. 417, 186–215 (2017)

    Article  Google Scholar 

  13. Alves, F., Pereira, A.I., Fernandes, A., Leitão, P.: Optimization of home care visits schedule by genetic algorithm. In: Korošec, P., Melab, N., Talbi, E.-G. (eds.) BIOMA 2018. LNCS, vol. 10835, pp. 1–12. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91641-5_1

    Chapter  Google Scholar 

  14. Eftimov, T., Petelin, G., Korošec, P.: DSCTool: a web-service-based framework for statistical comparison of stochastic optimization algorithms. Appl. Soft Comput. 87, 105977 (2020)

    Article  Google Scholar 

  15. Falcón-Cardona, J.G., Liefooghe, A., Coello Coello, C.A.: An ensemble indicator-based density estimator for evolutionary multi-objective optimization. In: Bäck, T., et al. (eds.) PPSN 2020. LNCS, vol. 12270, pp. 201–214. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58115-2_14

    Chapter  Google Scholar 

  16. Knowles, J., Thiele, L., Zitzler, E.: A tutorial on the performance assessment of stochastic multiobjective optimizers. Tik Rep. 214, 327–332 (2006)

    Google Scholar 

  17. Korošec, P., Eftimov, T.: Multi-objective optimization benchmarking using DSCTool. Mathematics 8(5), 839 (2020)

    Article  Google Scholar 

  18. Li, M., Yang, S., Liu, X.: Shift-based density estimation for pareto-based algorithms in many-objective optimization. IEEE Trans. Evol. Comput. 18(3), 348–365 (2014)

    Article  Google Scholar 

  19. Moubayed, N.A., Petrovski, A., McCall, J.A.W.: D\({}^{\text{2 }}\)MOPSO: MOPSO based on decomposition and dominance with archiving using crowding distance in objective and solution spaces. Evol. Comput. 22(1), 47–77 (2014)

    Article  Google Scholar 

  20. Riquelme, N., Von Lücken, C., Baran, B.: Performance metrics in multi-objective optimization. In: 2015 Latin American Computing Conference (CLEI), pp. 1–11. IEEE (2015)

    Google Scholar 

  21. Schütze, O.: A scalar optimization approach for averaged Hausdor approximations of the Pareto front. Eng. Optim. 48(9), 1593–1617 (2019)

    Article  Google Scholar 

  22. Székely, G.J., Rizzo, M.L.: Testing for equal distributions in high dimension. InterStat 5, 1–6 (2004)

    Google Scholar 

  23. Tian, Y., Cheng, R., Zhang, X., Jin, Y.: PlatEMO: a MATLAB platform for evolutionary multi-objective optimization [educational forum]. IEEE Comput. Intell. Mag. 12(4), 73–87 (2017)

    Article  Google Scholar 

  24. Zhang, Q., Li, H.: MOEA/D: a multi-objective evolutionary algorithm based on decomposition. IEEE Trans. Evol. Comput. 11(6), 712–731 (2007)

    Article  Google Scholar 

  25. Zitzler, E., Thiele, L.: Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Trans. Evol. Comput. 3(4), 257–271 (1999)

    Article  Google Scholar 

Download references

Acknowledgment

Our work was financially supported by the Slovenian Research Agency (research core funding No. P2-0098 and project No. Z2-1867). We also acknowledge support by COST Action CA15140 “Improving Applicability of Nature-Inspired Optimisation by Joining Theory and Practice (ImAppNIO)”.

Author information

Authors and Affiliations

  1. Leiden University, Leiden, The Netherlands

    Hao Wang

  2. Cinvestav-IPN, Mexico City, Mexico

    Carlos Igncio Hernández Castellanos

  3. Computer Systems Department, Jožef Stefan Institute, Ljubljana, Slovenia

    Tome Eftimov

Authors
  1. Hao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlos Igncio Hernández Castellanos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tome Eftimov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tome Eftimov .

Editor information

Editors and Affiliations

  1. Southern University of Science and Technology, Shenzhen, China

    Hisao Ishibuchi

  2. City University of Hong Kong, Kowloon Tong, China

    Qingfu Zhang

  3. Southern University of Science and Technology, Shenzhen, China

    Ran Cheng

  4. University of Exeter, Exeter, UK

    Ke Li

  5. Xi'an Jiaotong University, Xi'an, China

    Hui Li

  6. Xidian University, Xi'an, China

    Handing Wang

  7. East China Normal University, Shanghai, China

    Aimin Zhou

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, H., Castellanos, C.I.H., Eftimov, T. (2021). On Statistical Analysis of MOEAs with Multiple Performance Indicators. In: Ishibuchi, H., et al. Evolutionary Multi-Criterion Optimization. EMO 2021. Lecture Notes in Computer Science(), vol 12654. Springer, Cham. https://doi.org/10.1007/978-3-030-72062-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72062-9_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72061-2

  • Online ISBN: 978-3-030-72062-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation