Higher Order Mutation Testing to Drive Development of New Test Cases: An Empirical Comparison of Three Strategies

  • Quang Vu Nguyen
  • Lech Madeyski
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9621)


Mutation testing, which includes first order mutation (FOM) testing and higher order mutation (HOM) testing, appeared as a powerful and effective technique to evaluate the quality of test suites. The live mutants, which cannot be killed by the given test suite, make up a significant part of generated mutants and may drive the development of new test cases. Generating live higher order mutants (HOMs) able to drive development of new test cases is considered in this paper. We apply multi-objective optimization algorithms based on our proposed objectives and fitness functions to generate higher order mutants using three strategies: HOMT1 (HOMs generated from all first order mutants), HOMT2 (HOMs generated from killed first order mutants) and HOMT3 (HOMs generated from not-easy-to-kill first order mutants). We then use mutation score indicator to evaluate, which of the three approaches is better suited to drive development of new test cases and, as a result, to improve the software quality.


Mutation testing Higher order mutation testing Live mutants Equivalent mutants Multi-objective optimization algorithm 


  1. 1.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Comput. 11(4), 34–41 (1978)CrossRefGoogle Scholar
  2. 2.
    Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. SE-3(4), 279–290 (1977)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Jia, Y., Harman, M.: Higher order mutation testing. Inf. Softw. Technol. 51, 1379–1393 (2009)CrossRefGoogle Scholar
  4. 4.
    Harman, M., Jia, Y., Langdon, W.B.: A manifesto for higher order mutation testing. In: Third International Conference on Software Testing, Verification, and Validation Workshops, (2010)Google Scholar
  5. 5.
    Langdon, W.B., Harman, M., Jia, Y.: Efficient multi-objective higher order mutation testing with genetic programming. J. Syst. Softw. 83, 2416–2430 (2010)CrossRefGoogle Scholar
  6. 6.
    Jia, Y., Harman, M.: Constructing subtle faults using higher order mutation testing. In: Proceedings of the Eighth International Working Conference Source Code Analysis and Manipulation (2008)Google Scholar
  7. 7.
    Madeyski, L., Orzeszyna, W., Torkar, R., Józala, M.: Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014). doi: 10.1109/TSE.2013.44 CrossRefGoogle Scholar
  8. 8.
    Mresa, E.S., Bottaci, L.: Efficiency of mutation operators and selective mutation strategies: an empirical study. Softw. Test. Verification Reliab. 9(4), 205–232 (1999)CrossRefGoogle Scholar
  9. 9.
    Papadakis, M., Malevris, N.: An empirical evaluation of the first and second order mutation testing strategies. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, ser. ICSTW 2010, IEEE Computer Society, pp. 90–99 (2010)Google Scholar
  10. 10.
    Vincenzi, A.M.R., Nakagawa, E.Y., Maldonado, J.C., Delamaro, M.E., Romero, R.A.F.: Bayesian-learning based guidelines to determine equivalent mutants. Int. J. Softw. Eng. Knowl. Eng. 12(6), 675–690 (2002)CrossRefGoogle Scholar
  11. 11.
    Polo, M., Piattini, M., Garcia-Rodriguez, I.: Decreasing the cost of mutation testing with second-order mutants. Softw. Test. Verification Reliab. 19(2), 111–131 (2008)CrossRefGoogle Scholar
  12. 12.
    Nguyen, Q.V., Madeyski, L.: Problems of mutation testing and higher order mutation testing. In: van Do, T., Thi, H.A.L., Nguyen, N.T. (eds.) Advanced Computational Methods for Knowledge Engineering. AISC, vol. 282, pp. 157–172. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-06569-4_12 CrossRefGoogle Scholar
  13. 13.
    Zhu, H., Hall, P.A.V., May, J.H.R.: Software Unit Test Coverage and Adequacy. ACM Comput. Surv. 29(4), 366–427 (1997)CrossRefGoogle Scholar
  14. 14.
    Madeyski, L.: On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests. In: Münch, J., Abrahamsson, P. (eds.) PROFES 2007. LNCS, vol. 4589, pp. 207–221. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-73460-4_20 CrossRefGoogle Scholar
  15. 15.
    Madeyski, L.: The impact of pair programming on thoroughness and fault detection effectiveness of unit tests suites. Softw. Process: Improv. Pract. 13(3), 281–295 (2008). doi: 10.1002/spip.382 CrossRefGoogle Scholar
  16. 16.
    Madeyski, L.: The impact of test-first programming on branch coverage and mutation score indicator of unit tests: An experiment. Inf. Softw. Technol. 52(2), 169–184 (2010). doi: 10.1016/j.infsof.2009.08.007 CrossRefGoogle Scholar
  17. 17.
    Madeyski, L., Radyk, N.: Judy - a mutation testing tool for Java. IET Softw. 4(1), 32–42 (2010). doi: 10.1049/iet-sen.2008.0038 CrossRefGoogle Scholar
  18. 18.
    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multi objective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)CrossRefGoogle Scholar
  19. 19.
    Deb, K., Jain, H.: An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, Part I: solving problems with box constraints. IEEE Trans. Evol. Comput. 18(4), 577–601 (2014)CrossRefGoogle Scholar
  20. 20.
    Kollat, J.B., Reed, P.M., The value of online adaptive search: a performance comparison of NSGAII, ε-NSGAII and ε-MOEA. In: Coello Coello, C.A., Aguirre, A.H., Zitzler, E. (Eds.), Evolutionary Multi-Criterion Optimization, Third International Conference, EMO 2005 Guanajuato, Mexico, March 9–11 (2005)Google Scholar
  21. 21.
    Deb, K., Mohan, M., Mishra, S.: A fast multi-objective evolutionary algorithm for finding well-spread pareto-optimal solutions. KenGAL, Report No. 2003002. Indian Institute of Technology, Kanpur, India (2003)Google Scholar
  22. 22.
    Nguyen, Q.V., Madeyski, L.: Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm. In: Le Thi, H.A., Nguyen, N.T., Do, T.V. (eds.) Advanced Computational Methods for Knowledge Engineering. AISC, vol. 358, pp. 391–402. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-17996-4_35 Google Scholar
  23. 23.
    Purushothaman, R., Perry, D.E.: Toward Understanding the Rhetoric of small source code changes. IEEE Trans. Softw. Eng. 31(6), 511–526 (2005)CrossRefGoogle Scholar
  24. 24.
    Madeyski, L., Jureczko, M.: Which process metrics can significantly improve defect prediction models? An empirical study. Softw. Qual. J. 23(3), 393–422 (2015). doi: 10.1007/s11219-014-9241-7 CrossRefGoogle Scholar
  25. 25.
    Jureczko, M., Madeyski, L.: A review of process metrics in defect prediction studies. Metody Informatyki Stosowanej 30(5), 133–145 (2011). Google Scholar
  26. 26.
    Jureczko, M., Madeyski, L.: Towards identifying software project clusters with regard to defect prediction. In: Proceedings of the 6th International Conference on Predictive Models in Software Engineering (PROMISE 2010). ACM, New York, NY, USA, Article 9, 9:1–9:10. doi:  10.1145/1868328.1868342
  27. 27.
    Madeyski, L., Majchrzak, M.: Software measurement and defect prediction with DePress extensible framework. Found. Comput. Decis. Sci. 39(4), 249–270 (2014). doi: 10.2478/fcds-2014-0014 Google Scholar
  28. 28.
    Hryszko, J., Madeyski, L.: Bottlenecks in software defect prediction implementation in industrial projects. Found. Comput. Decis. Sci. 40(1), 17–33 (2015). doi: 10.1515/fcds-2015-0002 Google Scholar
  29. 29.
    Nguyen, Q.V., Madeyski, L.: Empirical evaluation of multi-objective optimization algorithms searching for higher order mutants. Cybern. Syst. Int. J. (Accepted 2016). DOI:  10.1080/01969722.2016.1128763 URL:

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Faculty of Computer Science and ManagementWroclaw University of TechnologyWroclawPoland

Personalised recommendations