A comparative study of many-objective evolutionary algorithms for the discovery of software architectures


During the design of complex systems, software architects have to deal with a tangle of abstract artefacts, measures and ideas to discover the most fitting underlying architecture. A common way to structure such complex systems is in terms of their interacting software components, whose composition and connections need to be properly adjusted. Along with the expected functionality, non-functional requirements are key at this stage to guide the many design alternatives to be evaluated by software architects. The appearance of Search Based Software Engineering (SBSE) brings an approach that supports the software engineer along the design process. Evolutionary algorithms can be applied to deal with the abstract and highly combinatorial optimisation problem of architecture discovery from a multiple objective perspective. The definition and resolution of many-objective optimisation problems is currently becoming an emerging challenge in SBSE, where the application of sophisticated techniques within the evolutionary computation field needs to be considered. In this paper, diverse non-functional requirements are selected to guide the evolutionary search, leading to the definition of several optimisation problems with up to 9 metrics concerning the architectural maintainability. An empirical study of the behaviour of 8 multi- and many-objective evolutionary algorithms is presented, where the quality and type of the returned solutions are analysed and discussed from the perspective of both the evolutionary performance and those aspects of interest to the expert. Results show how some many-objective evolutionary algorithms provide useful mechanisms to effectively explore design alternatives on highly dimensional objective spaces.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8


  1. 1.


  2. 2.


  3. 3.


  4. 4.


  5. 5.


  6. 6.


  7. 7.


  8. 8.

    Tables 10 (HV) and 11 (S) in Appendix show the results obtained by the Cliffs Delta test for the 9-objective combination. The full results in raw format for all the combinations are available at http://www.uco.es/grupos/kdis/sbse/RRV15


  1. Adra S, Fleming P (2011) Diversity management in evolutionary many-objective optimization. IEEE Trans Evol Comput 15(2):183–195

    Article  Google Scholar 

  2. Aleti A, Buhnova B, Grunske L, Koziolek A, Meedeniya I (2013) Software architecture optimization methods: a systematic literature review. IEEE Trans Softw Eng 39(5):658–683

    Article  Google Scholar 

  3. Arcuri A, Briand L (2011) A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of the 33rd international conference on software engineering (ICSE’11). IEEE, pp 1–10

  4. Assunção WKG, Colanzi TE, Vergilio SR, Pozo A (2014) A multi-objective optimization approach for the integration and test order problem. Inf Sci 267:119–139

    MathSciNet  Article  Google Scholar 

  5. Bader J, Zitzler E (2011) HypE: an algorithm for fast hypervolume-based many-objective optimization. Evol Comput 19(1):45–76

    Article  Google Scholar 

  6. Bansiya J, Davis CG (2002) A hierarchical model for object-oriented design quality assessment. IEEE Trans Softw Eng 28(1):4–17

    Article  Google Scholar 

  7. Bosch J, Molin P (1999) Software architecture design: evaluation and transformation. In: Proceedings of IEEE conference and workshop on engineering of computer-based systems (ECBS’99), pp 4–10

  8. Boussaïd I, Lepagnot J, Siarry P (2013) A survey on optimization metaheuristics. Inf Sci 237:82–117

    MathSciNet  Article  MATH  Google Scholar 

  9. Bouwers E, Correia J, van Deursen A, Visser J (2011) Quantifying the analyzability of software architectures. In: Proceedings of the 9th working IEEE/IFIP conference on software architecture (WICSA’11), pp 83–92

  10. Bowman M, Briand LC, Labiche Y (2010) Solving the class responsibility assignment problem in object-oriented analysis with multi-objective genetic algorithms. IEEE Trans Softw Eng 36 (6):817–837

    Article  Google Scholar 

  11. Coello Coello CA, Lamont GB, Van Veldhuizen DA (2007) Evolutionary algorithms for solving multi-objective problems, 2nd edn. Springer

  12. Deb K (2001) Multi-objective optimization using evolutionary algorithms. Wiley, New York

    Google Scholar 

  13. Deb K, Jain H (2014) An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, Part I: solving problems with box constraints. IEEE Trans Evol Comput 18(4):577–601

    Article  Google Scholar 

  14. Deb K, Mohan M, Mishra S (2003) Towards a quick computation of well-spread pareto-optimal solutions. In: Evolutionary multi-criterion optimization of LNCS, vol 2632. Springer, pp 222–236

  15. Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6(2):182–197

    Article  Google Scholar 

  16. del Sagrado J, del Águila IM, Orellana FJ (2015) Multi-objective ant colony optimization for requirements selection. Empir Softw Eng 20(3):577–610

    Article  Google Scholar 

  17. Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol Comput 1(1):3–18

    Article  Google Scholar 

  18. Dobrica L, Niemela E (2002) A survey on software architecture analysis methods. IEEE Trans Softw Eng 28(7):638–653

    Article  MATH  Google Scholar 

  19. Ducasse S, Pollet D (2009) Software architecture reconstruction: a process-oriented taxonomy. IEEE Trans Softw Eng 35(4):573–591

    Article  Google Scholar 

  20. Durillo JJ, Zhang Y, Alba E, Harman M, Nebro AJ (2011) A study of the bi-objective next release problem. Empir Softw Eng 16(1):29–60

    Article  Google Scholar 

  21. Garlan D (2000) Software architecture: a roadmap. In: Proceedings of the 22th international conference of software engineering (ICSE’00), pp 91–101

  22. Grunske L (2006) Identifying ”Good” architectural design alternatives with multi-objective optimization strategies. In: Proceedings of the 28th international conference on software engineering (ICSE’06), pp 849–852

  23. Gupta P, Verma S, Mehlawat M (2012) Optimization model of COTS selection based on cohesion and coupling for modular software systems under multiple applications environment. In: Computational science and its applications (ICCSA) of LNCS, vol 7335. Springer, pp 87–102

  24. Hadka D, Reed P (2013) Borg: an auto-adaptive many-objective evolutionary computing framework. Evol Comput 21(2):231–259

    Article  Google Scholar 

  25. Harman M, Mansouri SA, Zhang Y (2012) Search-based software engineering: trends, techniques and applications. ACM Comput Surv 45(1):11:1–61

    Article  Google Scholar 

  26. He Z, Yen G, Zhang J (2014) Fuzzy-based Pareto optimality for many-objective evolutionary algorithms. IEEE Trans Evol Comput 18(2):269–285

    Article  Google Scholar 

  27. ISO (2011a) ISO/IEC 25010:2011(E). Software product Quality Requirements and Evaluation (SQuaRE) - System and software quality models. ISO

  28. ISO (2011b) ISO/IEC/IEEE FDIS 42010/D9. Systems and software engineering - Architecture description

  29. Kalboussi S, Bechikh S, Kessentini M, Ben Said L (2013) Preference-based many-objective evolutionary testing generates harder test cases for autonomous agents. In: Proceedings of 5th symposium on search based software engineering (SSBSE’13), pp 245–250

  30. Khare V, Yao X, Deb K (2003) Performance scaling of multi-objective evolutionary algorithms. In: Evolutionary multi-criterion optimization of lecture notes in computer science, vol 2632. Springer, Berlin, pp 376–390

    Google Scholar 

  31. Koziolek A, Ardagna D, Mirandola R (2013) Hybrid multi-attribute QoS optimization in component based software systems. J Syst Softw 86(10):2542–2558

    Article  Google Scholar 

  32. Krogmann K (2010) Reconstruction of software component architectures and behaviour models using static and dynamic analysis. KIT Scientific Publishing

  33. Li R, Etemaadi R, Emmerich MTM, Chaudron MRV (2011) An evolutionary multiobjective optimization approach to component-based software architecture design. In: Proceedings of the IEEE congress on evolutionary computation (CEC’11), pp 432–439

  34. Luna F, González-Álvarez DL, Chicano F, Vega-Rodríguez MA (2014) The software project scheduling problem: A scalability analysis of multi-objective metaheuristics. Appl Soft Comput 15 :136–148

    Article  Google Scholar 

  35. Lutz R (2001) Evolving good hierarchical decompositions of complex systems. J Syst Archit 47(7):613–634

    Article  Google Scholar 

  36. Martin R (1994) OO design quality metrics - An analysis of dependencies. In: Object-Oriented programming systems, languages and applications (OOPSLA), pp 1–8

  37. Mkaouer MW, Kessentini M, Bechikh S, Deb K, Ó Cinnéide M (2014) High dimensional search-based software engineering: finding tradeoffs among 15 objectives for automating software refactoring using NSGA-III. In: Proceedings of the 16th annual genetic and evolutionary computation conference (GECCO’14), pp 1263–1270

  38. Narasimhan VL, Hendradjaya B (2007) Some theoretical considerations for a suite of metrics for the integration of software components. Inf Sci 177(3):844–864

    MathSciNet  Article  MATH  Google Scholar 

  39. OMG (2010) Unified modeling language 2.4 superstructure specification. OMG. formal/2010-11-14, http://www.omg.org/spec/UML/2.4/

  40. Ouni A, Kessentini M, Sahraoui H, Hamdi MS (2013) The use of development history in software refactoring using a multi-objective evolutionary algorithm. In: Proceedings of the 15th annual genetic and evolutionary computation conference (GECCO’13), pp 1461–1468

  41. Praditwong K, Harman M, Yao X (2011) Software module clustering as a multi-objective search problem. IEEE Trans Softw Eng 37(2):264–282

    Article  Google Scholar 

  42. Praditwong K, Yao X (2007) How well do multi-objective evolutionary algorithms scale to large problems. In: Proceedings of the IEEE congress on evolutionary computation (CEC’07), pp 3959–3966

  43. Purshouse R, Fleming P (2007) On the evolutionary optimization of many conflicting objectives. IEEE Trans Evol Comput 11(6):770–784

    Article  Google Scholar 

  44. Räihä O (2010) A survey on search-based software design. Comput Sci Rev 4 (4):203–249

    Article  Google Scholar 

  45. Räihä O, Koskimies K, Makinen E (2011) Generating software architecture spectrum with multi-objective genetic algorithms. In: Proceedings of the 3th world congress on nature and biologically inspired computing (NaBIC’11), pp 29–36

  46. Ramírez A, Romero JR, Ventura S (2014) On the performance of multiple objective evolutionary algorithms for software architecture discovery. In: Proceedings of the 16th annual genetic and evolutionary computation conference (GECCO’14), pp 1287–1294

  47. Ramírez A, Romero JR, Ventura S (2015) An approach for the evolutionary discovery of software architectures. Inf Sci 305:234–255

    Article  Google Scholar 

  48. Romano J, Kromrey JD, Coraggio J, Showronek J (2006) Appropriate statistics for ordinal level data: should we really be using t-test and cohen’s d for evaluating group differences on the nsse and other surveys? In: Annual meeting of the Florida association of institutional research

  49. Sant’Anna C, Figueiredo E, Garcia A, Lucena CJ (2007) On the modularity of software architectures: a concern-driven measurement framework. In: Software architecture of LNCS, vol 4758. Springer, pp 207–224

  50. Sayyad A, Ammar H (2013) Pareto-optimal search-based software engineering (POSBSE): a literature survey. In: 2nd inter. workshop on realizing artificial intelligence synergies in software engineering (RAISE), pp 21–27

  51. Sayyad AS, Menzies T, Ammar H (2013) On the value of user preferences in search-based software engineering: a case study in software product lines. In: Proceedings of the 35th international conference on software engineering (ICSE’13), pp 492–501

  52. Schutze O, Lara A, Coello Coello CA (2011) On the influence of the number of objectives on the hardness of a multiobjective optimization problem. IEEE Trans Evol Comput 15(4):444–455

    Article  Google Scholar 

  53. Szyperski C (2002) Component software: beyond object-oriented programming, 2nd edn. Addison-Wesley, Boston

    Google Scholar 

  54. Ventura S, Romero C, Zafra A, Delgado JA, Hervás C (2008) JCLEC: a Java framework for evolutionary computation. Soft Comput 12(4):381–392

    Article  Google Scholar 

  55. von Lücken C, Barán B, Brizuela C (2014) A survey on multi-objective evolutionary algorithms for many-objective problems. Comput Optim Appl 58(3):707–756

    MathSciNet  MATH  Google Scholar 

  56. Wagner T, Beume N, Naujoks B (2007) Pareto-, aggregation-, and indicator-based methods in many-objective optimization. In: Evolutionary multi-criterion optimization of LNCS, vol 4403. Springer, pp 742–756

  57. Wang H, Jiao L, Yao X (2014) An improved two-archive algorithm for many-objective optimization. IEEE Trans Evol Comput. To appear

  58. Yang S, Li M, Liu X, Zheng J (2013) A grid-based evolutionary algorithm for many-objective optimization. IEEE Trans Evol Comput 17(5):721–736

    Article  Google Scholar 

  59. Yao X (2013) Some recent work on multi-objective approaches to search-based software engineering. In: Proceedings of the 5th symposium on search based software engineering (SSBSE), pp 4–15

  60. Zhang Q, Li H (2007) MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans Evol Comput 11(6):712–731

    Article  Google Scholar 

  61. Zhang Y, Harman M, Lim SL (2013) Empirical evaluation of search based requirements interaction management. Inf Softw Technol 55(1):126–152

    Article  Google Scholar 

  62. Zhou A, Qu B-Y, Li H, Zhao S-Z, Suganthan PN, Zhang Q (2011) Multiobjective evolutionary algorithms: a survey of the state of the art. Swarm Evol Comput 1(1):32–49

    Article  Google Scholar 

  63. Zitzler E, Künzli S (2004) Indicator-based selection in multiobjective search. In: Parallel problem solving from nature - PPSN VIII of LNCS, vol 3242. Springer, pp 832–842

  64. Zitzler E, Laumanns M, Bleuler S (2004) A tutorial on evolutionary multiobjective optimization. In: Metaheuristics for multiobjective optimisation of lecture notes in economics and mathematical systems, vol 535. Springer, Berlin, pp 3–37

    Google Scholar 

  65. Zitzler E, Laumanns M, Thiele L (2001) SPEA2: improving the strength Pareto evolutionary algorithm. In: Proceedings of the conference on evolutionary methods for design, optimisation and control with applications to industrial problems, pp 95–100

Download references


Work supported by the Spanish Ministry of Science and Technology, project TIN2011-22408, the Spanish Ministry of Economy and Competitiveness, project TIN2014-55252-P, and FEDER funds. This research was also supported by the Spanish Ministry of Education under the FPU program (FPU13/01466).

Author information



Corresponding author

Correspondence to José Raúl Romero.

Additional information

Communicated by: Marouane Kessentini and Guenther Ruhe



Tables 10 and 11 present the results of the Cliff’s Delta test for hypervolume (HV) and spacing (S), respectively. The value of a cell, x i, j , represents the effect size and its interpretation when comparing the algorithm i against the algorithm j for the 9-objective optimisation problem in terms of the specific quality indicator. Table 12 shows the results of the Friedman and the Holm tests for all possible combinations of 2 objectives. Tables 13, 14, 15, 16 report the results for the 4-objective problems, whereas the 6-objective problems are shown from Table 17, 18, 19. In addition, Table 19 contains the results obtained from 8-objective and 9-objective problems. For each of them, the best rankings for the two quality indicators, HV and S, are shown in bold typeface, and their cells are shaded in gray colour when significant differences exist. The critical value, according to the F-Distribution with 6 and 54 degrees of freedom, i.e. the p-value, is 2.2720. Since the Holm test is performed if H 0 is rejected, Tables 1219 show these ranking values in italic typeface when the corresponding algorithm lies below its critical threshold, i.e. its performance according to the column-specific indicator is worse than that provided by the best algorithm.

Table 10 Results of the Cliff’s Delta test for hypervolume (n = negligible, s = small, m = medium, l = large) (α = 0.05)
Table 11 Results of the Cliff’s Delta test for spacing (n = negligible, s = small, m = medium, l = large) (α = 0.05)
Table 12 Average rankings for 2-objective problems obtained from the Friedman test (α = 0.05)
Table 13 Average rankings for 4-objective problems obtained from the Friedman Test (α = 0.05) (cont’d)
Table 14 Average rankings for 4-objective problems obtained from the Friedman Test (α = 0.05) (cont’d)
Table 15 Average rankings for 4-objective problems obtained from the Friedman Test (α = 0.05) (cont’d)
Table 16 Average rankings for 4-objective problems obtained from the Friedman test (α = 0.05)
Table 17 Average rankings for 6-objective problems obtained from the Friedman Test (α = 0.05) (cont’d)
Table 18 Average rankings for 6-objective problems obtained from the Friedman Test (α = 0.05) (cont’d)
Table 19 Average rankings for 6-, 8- and 9-objective problems obtained from the Friedman Test (α = 0.05)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ramírez, A., Romero, J.R. & Ventura, S. A comparative study of many-objective evolutionary algorithms for the discovery of software architectures. Empir Software Eng 21, 2546–2600 (2016). https://doi.org/10.1007/s10664-015-9399-z

Download citation


  • Software architecture discovery
  • Search based software engineering
  • Many-objective evolutionary algorithms
  • Multi-objective evolutionary algorithms