Arabian Journal for Science and Engineering

, Volume 43, Issue 12, pp 7473–7486 | Cite as

Reducing the Cost of Higher-Order Mutation Testing

  • Ahmed S. GhidukEmail author
  • Moheb R. Girgis
  • Marwa H. Shehata
Research Article - Computer Engineering and Computer Science


Constructing mutants of order higher than first order is a key step in higher-order mutation testing. The majority of higher-order mutant generation techniques merge two (or more) first-order mutants (FOMs) to build a higher-order mutant. Unfortunately, these techniques suffer from the high cost due to the explosion in the number of higher-order mutants (HOMs). Consequently, developing techniques to find the minimum adequate and effective number of mutants are desired. Earlier work reduced the number of mutants by considering only a subset of mutants, a subset of operators, or selecting specific locations in the program (to be mutated) instead of the whole program. In this paper, we present three new techniques (SCWR, 2E2O, and 2E2OWR) to generate and reduce the overall number of HOMs and the equivalent ones as well. Each technique merges FOMs in more effective and simpler way than the previous techniques to find more effective HOMs. These techniques have been applied on a benchmark of programs, and the results have been compared to the results of some related work such as DiffOp, JudyDiffOp, and Last2First techniques. The results showed that SCWR, 2E2O, and 2E2OWR outperformed the related work and reduced the total number of mutants by 67.9, 16.0, and 60.1% comparing to approximately 50% for the related work and the number of equivalent mutants by 66.9, 79.6, and 65.0% comparing to 25.8 and 36.4% for the related work, respectively.


Mutation testing Higher-order mutation testing Mutant generation First- and higher-order mutants 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    DeMillo, R.A.; Lipton, R.J.; Sayward, F.G.: Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41 (1978)CrossRefGoogle Scholar
  2. 2.
    Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. 3(4), 279–290 (1977)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Polo, M.; Piattini, M.; Garcia-Rodriguez, I.: Decreasing the cost of mutation testing with second-order mutants. Softw. Test. Verif. Reliab. 19(2), 111–131 (2008)CrossRefGoogle Scholar
  4. 4.
    Kintis, M.; Papadakis, M.; Malevris, N.: Evaluating mutation testing alternatives: a collateral experiment. In: Proceedings of 17th Asia Pacific Software Engineering Conference (APSEC) (2010)Google Scholar
  5. 5.
    Jia, Y.; Harman, M.: Higher order mutation testing. J. Inf. Softw. Technol. 51(10), 1379–1393 (2009)CrossRefGoogle Scholar
  6. 6.
    Offutt, A.J.: Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1, 5–20 (1992)CrossRefGoogle Scholar
  7. 7.
    Papadakis, M.; Malevris, N.: An empirical evaluation of the first and second order mutation testing strategies. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, Series. ICSTW’10, IEEE Computer Society, pp. 90–99 (2010)Google Scholar
  8. 8.
    Madeyski, L.; Orzeszyna, W.; Torkar, R.; Józala, M.: Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014)CrossRefGoogle Scholar
  9. 9.
    Langdon, WB.; Harman, M.; Jia, Y.: Multi objective mutation testing with genetic programming. In: 4th Testing Academia and Industry Conference—Practice and Research Techniques (TAIC PART’09), Windsor, UK, IEEE press, pp. 21–29 (2009)Google Scholar
  10. 10.
    Harman, M.; Jia, Y.; Langdon, W.B.: Strong higher order mutation-based test data generation. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE’11), pp. 212–222 (2011)Google Scholar
  11. 11.
    Jia, Y.; Harman, M.: Constructing subtle faults using higher order mutation testing. In: Proceedings of the 8th International Working Conference on Source Code Analysis and Manipulation (SCAM’08), Beijing, China, pp. 249–258 (2009)Google Scholar
  12. 12.
    Nguyen, Q.V.; Madeyski, L.: Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm. In: Le H.A.; Thi, N.; Nguyen, T.; Do, T.V. (eds.) Advanced Computational Methods for Knowledge Engineering. Advances in Intelligent Systems and Computing, vol. 358, Springer, pp. 391–402 (2015). Google Scholar
  13. 13.
    Nguyen, Q.V.; Madeyski, L.: Empirical evaluation of multi-objective optimization algorithms searching for higher order mutants. In: Cybernetics and Systems—Smart Experience and Knowledge Engineering for Optimization, Learning, and Classification/Recommendation Problems, vol. 47, pp. 48–68 (2016)CrossRefGoogle Scholar
  14. 14.
    Nguyen, Q.V.; Madeyski, L.: Higher order mutation testing to drive development of new test cases: an empirical comparison of three strategies. In: Nguyen, N.T.; Trawiński, B.; Fujita, H.; Hong, T.P (eds.) Intelligent Information and Database Systems (ACIIDS 2016), Lecture Notes in Artificial Intelligence, vol. 9621. Springer, pp. 235–244 (2016). CrossRefGoogle Scholar
  15. 15.
    Wong, W.E.; Mathur, A.P.: Reducing the cost of mutation testing: an empirical study. J. Syst. Softw. 31(3), 185–196 (1995)CrossRefGoogle Scholar
  16. 16.
    Acree, A.T.: On mutation. Ph.D. Dissertation, Georgia Inst of Tech Atlanta School of Information and Computer Science (1980)Google Scholar
  17. 17.
    Budd, T.A.: Mutation analysis of program test data. Ph.D. Dissertation, Yale University (1980)Google Scholar
  18. 18.
    Hussain, S.: Mutation clustering. Master’s Thesis, King’s College London (2008)Google Scholar
  19. 19.
    Offutt, A.J.; Rothermel, G.; Zapf, C.: An experimental evaluation of selective mutation. In: Proceedings of 15th International Conference on Software Engineering, pp. 100–107 (1993)Google Scholar
  20. 20.
    Ghiduk, A.S.: Reducing the number of higher-order mutants with the aid of data flow e-informatica. Softw. Eng. J. 10, 31–49 (2016)Google Scholar
  21. 21.
    Ma, Y.-S.; Offutt, A.J.; Kwon, Y.-R.: MuJava: an automated class mutation system. Softw. Test. Verif. Reliab. 15(2), 97–133 (2005)CrossRefGoogle Scholar
  22. 22.
    Jia, Y.; Harman, M.: MILU: a customizable, runtime-optimized higher order mutation testing tool for the full C language. In: Proceedings of the 3rd Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART’08), Windsor, UK: IEEE Computer Society, pp. 94–98 (2008)Google Scholar
  23. 23.
    MuClipse. Internet (2008)
  24. 24.
    Kintis, M.; Papadakis, M.; Malevris, N.: Isolating first order equivalent mutants via second order mutation. In: IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 701–710 (2012)Google Scholar
  25. 25.
    Ghiduk, A.S.: Using evolutionary algorithms for higher-order mutation testing. IJCSI Int. J. Comput. Sci. 11(2), 93–104 (2014)Google Scholar
  26. 26.
    Allen, F.E.; Cocke, J.: A program data flow analysis procedure. Commun. ACM 19(3), 137 (1976) [Online]. CrossRefGoogle Scholar
  27. 27.
    Ghiduk, A.S.; Girgis, M.R.; Hashim, M.: Higher-order mutation testing: a systematic literature review. Int. J. Comput. Sci. Rev. 25, 29–48 (2017)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Papadakis, M.; Henard, C.; Harman, M.; Jia, Y.; Traon, Y.L.: Threats to the validity of mutation-based test assessment. In: Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016), pp. 354–365 (2016)Google Scholar
  29. 29.
    Umar, M.: An Evaluation of Mutation Operators for Equivalent Mutants. MS Project, Kings College, London (2006)Google Scholar
  30. 30.
    Harman, M.; Jia, Y.; Langdon, W.B.: Strong higher order mutation-based test data generation. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11), pp. 212–222 (2011)Google Scholar
  31. 31.
    Chekam, TT.; Papadakis, M.; Traon, YL.; Harman, M.: An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In: Proceedings of the 39th International Conference on Software Engineering (ICSE ’17), pp. 597–608 (2017)Google Scholar

Copyright information

© King Fahd University of Petroleum & Minerals 2018

Authors and Affiliations

  • Ahmed S. Ghiduk
    • 1
    • 2
    Email author
  • Moheb R. Girgis
    • 3
  • Marwa H. Shehata
    • 1
  1. 1.College of Computers and ITTaif UniversityTaifSaudi Arabia
  2. 2.Department of Mathematics and Computer Science, Faculty of ScienceBeni-Suef UniversityBeni-SuefEgypt
  3. 3.Department of Computer Science, Faculty of ScienceMinia UniversityMinyaEgypt

Personalised recommendations