On the Relationship Between the Order of Mutation Testing and the Properties of Generated Higher Order Mutants

  • Quang Vu Nguyen
  • Lech Madeyski
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9621)


The goal of higher order mutation testing is to improve mutation testing effectiveness in particular and test effectiveness in general. There are different approaches which have been proposed in the area of second order mutation testing and higher order mutation testing with mutants order ranging from 2 to 70. Unfortunately, the empirical evidence on the relationship between the order of mutation testing and the desired properties of generated mutants is scarce except the conviction that the number of generated mutants could grow exponentially with the order of mutation testing. In this paper, we present the study of finding the relationships between the order of mutation testing and the properties of mutants in terms of number of generated high quality and reasonable mutants as well as generated live mutants. Our approach includes higher order mutants classification, objective functions and fitness functions to classify and identify generated higher order mutants. We use four multi-objective optimization algorithms for constructing higher order mutants. Obtained empirical results indicate that 5 is a relevant highest order in higher order mutation testing.


Mutation testing Higher order mutation Higher order mutants Multi-objective optimization algorithm 


  1. 1.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Comput. 11(4), 34–41 (1978)CrossRefGoogle Scholar
  2. 2.
    Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Trans. Software Eng. 3(4), 279–290 (1977)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Harman, M., Jia, Y., Langdon, W.B.: A manifesto for higher order mutation testing. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops. ICSTW 2010, pp. 80–89. IEEE Computer Society, Washington, DC (2010)Google Scholar
  4. 4.
    Jia, Y., Harman, M.: Constructing subtle faults using higher order mutation testing. In: Source Code Analysis and Manipulation. pp. 249–258 (2008)Google Scholar
  5. 5.
    Jia, Y., Harman, M.: Higher order mutation testing. Inf. Softw. Technol. 51(10), 1379–1393 (2009)CrossRefGoogle Scholar
  6. 6.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)CrossRefGoogle Scholar
  7. 7.
    Kintis, M., Papadakis, M., Malevris, N.: Evaluating mutation testing alternatives: a collateral experiment. In: 2010 17th Asia Pacific Software Engineering Conference (APSEC), pp. 300–309, November 2010Google Scholar
  8. 8.
    Langdon, W.B., Harman, M., Jia, Y.: Efficient multi-objective higher order mutation testing with genetic programming. J. Syst. Softw. 83(12), 2416–2430 (2010)CrossRefGoogle Scholar
  9. 9.
    Madeyski, L.: On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests. In: Münch, J., Abrahamsson, P. (eds.) PROFES 2007. LNCS, vol. 4589, pp. 207–221. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-73460-4_20 CrossRefGoogle Scholar
  10. 10.
    Madeyski, L.: Impact of pair programming on thoroughness and fault detection effectiveness of unit test suites. Softw. Process Improv. Pract. 13(3), 281–295 (2008). doi: 10.1002/spip.382. CrossRefGoogle Scholar
  11. 11.
    Madeyski, L.: The impact of test-first programming on branch coverage and mutation score indicator of unit tests: an experiment. Inf. Softw. Technol. 52(2), 169–184 (2010). doi: 10.1016/j.infsof.2009.08.007 CrossRefGoogle Scholar
  12. 12.
    Madeyski, L., Orzeszyna, W., Torkar, R., Józala, M.: Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014). doi: 10.1109/TSE.2013.44 CrossRefGoogle Scholar
  13. 13.
    Madeyski, L., Radyk, N.: Judy - a mutation testing tool for java. IET Softw. 4(1), 32–42 (2010). doi: 10.1049/iet-sen.2008.0038 CrossRefGoogle Scholar
  14. 14.
    Nguyen, Q.V., Madeyski, L.: Problems of mutation testing and higher order mutation testing. In: van Do, T., Thi, H.A.L., Nguyen, N.T. (eds.) Advanced Computational Methods for Knowledge Engineering. AISC, vol. 282, pp. 157–172. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-06569-4_12 CrossRefGoogle Scholar
  15. 15.
    Nguyen, Q.V., Madeyski, L.: Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm. In: Le Thi, H.A., Nguyen, N.T., Do, T.V. (eds.) Advanced Computational Methods for Knowledge Engineering. AISC, vol. 358, pp. 391–402. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-17996-4_35 Google Scholar
  16. 16.
    Nguyen, Q.V., Madeyski, L.: Empirical evaluation of multi-objective optimization algorithms searching for higher order mutants. An International Journal on Cybernetics andSystems (2016). doi: 10.1080/01969722.2016.1128763,
  17. 17.
    Omar, E., Ghosh, S., Whitley, D.: Constructing subtle higher order mutants for Java and aspectJ programs. In: 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), pp. 340–349, November 2013Google Scholar
  18. 18.
    Papadakis, M., Malevris, N.: An empirical evaluation of the first and second order mutation testing strategies. In: 2010 Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), pp. 90–99, April 2010Google Scholar
  19. 19.
    Polo, M., Piattini, M., García-Rodríguez, I.: Decreasing the cost of mutation testing with second-order mutants. Softw. Test. Verification Reliab. 19(2), 111–131 (2009). doi: 10.1002/stvr.392 CrossRefGoogle Scholar
  20. 20.
    Purushothaman, R., Perry, D.E.: Toward understanding the rhetoric of small source code changes. IEEE Trans. Softw. Eng. 31(6), 511–526 (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Faculty of Computer Science and ManagementWroclaw University of TechnologyWroclawPoland

Personalised recommendations