Many Independent Objective (MIO) Algorithm for Test Suite Generation

  • Andrea Arcuri
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10452)


Automatically generating test suites is intrinsically a multi-objective problem, as any of the testing targets (e.g., statements to execute or mutants to kill) is an objective on its own. Test suite generation has peculiarities that are quite different from other more regular optimisation problems. For example, given an existing test suite, one can add more tests to cover the remaining objectives. One would like the smallest number of small tests to cover as many objectives as possible, but that is a secondary goal compared to covering those targets in the first place. Furthermore, the amount of objectives in software testing can quickly become unmanageable, in the order of (tens/hundreds of) thousands, especially for system testing of industrial size systems. Traditional multi-objective optimisation algorithms can already start to struggle with just four or five objectives to optimize. To overcome these issues, different techniques have been proposed, like for example the Whole Test Suite (WTS) approach and the Many-Objective Sorting Algorithm (MOSA). However, those techniques might not scale well to very large numbers of objectives and limited search budgets (a typical case in system testing). In this paper, we propose a novel algorithm, called Many Independent Objective (MIO) algorithm. This algorithm is designed and tailored based on the specific properties of test suite generation. An empirical study, on a set of artificial and actual software, shows that the MIO algorithm can achieve higher coverage compared to WTS and MOSA, as it can better exploit the peculiarities of test suite generation.


Test generation SBSE SBST MOO 



This work is supported by the National Research Fund, Luxembourg (FNR/P10/03).


  1. 1.
    Ali, S., Iqbal, M.Z., Arcuri, A., Briand, L.C.: Generating test data from OCL constraints with search techniques. IEEE Trans. Softw. Eng. (TSE) 39(10), 1376–1402 (2013)CrossRefGoogle Scholar
  2. 2.
    Arcuri, A., Briand, L.: Adaptive random testing: an illusion of effectiveness? In: ACM International Symposium on Software Testing and Analysis (ISSTA), pp. 265–275 (2011)Google Scholar
  3. 3.
    Arcuri, A., Briand, L.: A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verification Reliab. (STVR) 24(3), 219–250 (2014)CrossRefGoogle Scholar
  4. 4.
    Arcuri, A., Fraser, G.: Parameter tuning or default values? an empirical investigation in search-based software engineering. Empirical Softw. Eng. (EMSE) 18(3), 594–623 (2013)CrossRefGoogle Scholar
  5. 5.
    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. (TEVC) 6(2), 182–197 (2002)CrossRefGoogle Scholar
  6. 6.
    Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39(2), 276–291 (2013)CrossRefGoogle Scholar
  7. 7.
    Fraser, G., Arcuri, A.: Achieving scalable mutation-based generation of whole test suites. Empirical Softw. Eng. (EMSE) 20(3), 783–812 (2015)CrossRefGoogle Scholar
  8. 8.
    Harman, M., McMinn., P.: A theoretical and empirical study of search based testing: local, global and hybrid search. IEEE Trans. Softw. Eng. (TSE) 36(2), 226–247 (2010)Google Scholar
  9. 9.
    Harman, M., Mansouri, S.A., Zhang, Y.: Search-based software engineering: trends, techniques and applications. ACM Comput. Surv. (CSUR) 45(1), 11 (2012)CrossRefGoogle Scholar
  10. 10.
    Li, B., Li, J., Tang, K., Yao, X.: Many-objective evolutionary algorithms: a survey. ACM Comput. Surv. (CSUR) 48(1), 13 (2015)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Panichella, A., Kifetew, F., Tonella, P.: Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans. Softw. Eng. (TSE) PP(99), 1 (2017).
  12. 12.
    Rojas, J.M., Vivanti, M., Arcuri, A., Fraser, G.: A detailed investigation of the effectiveness of whole test suite generation. Empirical Softw. Eng. (EMSE) 22(2), 852–893 (2017).
  13. 13.
    Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Faculty of TechnologyWesterdals Oslo ACTOsloNorway
  2. 2.University of LuxembourgLuxembourg CityLuxembourg

Personalised recommendations