Skip to main content
SpringerLink
Log in

Reducing the Cost of Higher-Order Mutation Testing

  • Research Article - Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

Constructing mutants of order higher than first order is a key step in higher-order mutation testing. The majority of higher-order mutant generation techniques merge two (or more) first-order mutants (FOMs) to build a higher-order mutant. Unfortunately, these techniques suffer from the high cost due to the explosion in the number of higher-order mutants (HOMs). Consequently, developing techniques to find the minimum adequate and effective number of mutants are desired. Earlier work reduced the number of mutants by considering only a subset of mutants, a subset of operators, or selecting specific locations in the program (to be mutated) instead of the whole program. In this paper, we present three new techniques (SCWR, 2E2O, and 2E2OWR) to generate and reduce the overall number of HOMs and the equivalent ones as well. Each technique merges FOMs in more effective and simpler way than the previous techniques to find more effective HOMs. These techniques have been applied on a benchmark of programs, and the results have been compared to the results of some related work such as DiffOp, JudyDiffOp, and Last2First techniques. The results showed that SCWR, 2E2O, and 2E2OWR outperformed the related work and reduced the total number of mutants by 67.9, 16.0, and 60.1% comparing to approximately 50% for the related work and the number of equivalent mutants by 66.9, 79.6, and 65.0% comparing to 25.8 and 36.4% for the related work, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. DeMillo, R.A.; Lipton, R.J.; Sayward, F.G.: Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41 (1978)

    Article  Google Scholar 

  2. Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. 3(4), 279–290 (1977)

    Article  MathSciNet  Google Scholar 

  3. Polo, M.; Piattini, M.; Garcia-Rodriguez, I.: Decreasing the cost of mutation testing with second-order mutants. Softw. Test. Verif. Reliab. 19(2), 111–131 (2008)

    Article  Google Scholar 

  4. Kintis, M.; Papadakis, M.; Malevris, N.: Evaluating mutation testing alternatives: a collateral experiment. In: Proceedings of 17th Asia Pacific Software Engineering Conference (APSEC) (2010)

  5. Jia, Y.; Harman, M.: Higher order mutation testing. J. Inf. Softw. Technol. 51(10), 1379–1393 (2009)

    Article  Google Scholar 

  6. Offutt, A.J.: Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1, 5–20 (1992)

    Article  Google Scholar 

  7. Papadakis, M.; Malevris, N.: An empirical evaluation of the first and second order mutation testing strategies. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, Series. ICSTW’10, IEEE Computer Society, pp. 90–99 (2010)

  8. Madeyski, L.; Orzeszyna, W.; Torkar, R.; Józala, M.: Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014)

    Article  Google Scholar 

  9. Langdon, WB.; Harman, M.; Jia, Y.: Multi objective mutation testing with genetic programming. In: 4th Testing Academia and Industry Conference—Practice and Research Techniques (TAIC PART’09), Windsor, UK, IEEE press, pp. 21–29 (2009)

  10. Harman, M.; Jia, Y.; Langdon, W.B.: Strong higher order mutation-based test data generation. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE’11), pp. 212–222 (2011)

  11. Jia, Y.; Harman, M.: Constructing subtle faults using higher order mutation testing. In: Proceedings of the 8th International Working Conference on Source Code Analysis and Manipulation (SCAM’08), Beijing, China, pp. 249–258 (2009)

  12. Nguyen, Q.V.; Madeyski, L.: Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm. In: Le H.A.; Thi, N.; Nguyen, T.; Do, T.V. (eds.) Advanced Computational Methods for Knowledge Engineering. Advances in Intelligent Systems and Computing, vol. 358, Springer, pp. 391–402 (2015). https://doi.org/10.1007/978-3-319-17996-4

    Google Scholar 

  13. Nguyen, Q.V.; Madeyski, L.: Empirical evaluation of multi-objective optimization algorithms searching for higher order mutants. In: Cybernetics and Systems—Smart Experience and Knowledge Engineering for Optimization, Learning, and Classification/Recommendation Problems, vol. 47, pp. 48–68 (2016)

    Article  Google Scholar 

  14. Nguyen, Q.V.; Madeyski, L.: Higher order mutation testing to drive development of new test cases: an empirical comparison of three strategies. In: Nguyen, N.T.; Trawiński, B.; Fujita, H.; Hong, T.P (eds.) Intelligent Information and Database Systems (ACIIDS 2016), Lecture Notes in Artificial Intelligence, vol. 9621. Springer, pp. 235–244 (2016). https://doi.org/10.1007/978-3-662-49381-6_23

    Chapter  Google Scholar 

  15. Wong, W.E.; Mathur, A.P.: Reducing the cost of mutation testing: an empirical study. J. Syst. Softw. 31(3), 185–196 (1995)

    Article  Google Scholar 

  16. Acree, A.T.: On mutation. Ph.D. Dissertation, Georgia Inst of Tech Atlanta School of Information and Computer Science (1980)

  17. Budd, T.A.: Mutation analysis of program test data. Ph.D. Dissertation, Yale University (1980)

  18. Hussain, S.: Mutation clustering. Master’s Thesis, King’s College London (2008)

  19. Offutt, A.J.; Rothermel, G.; Zapf, C.: An experimental evaluation of selective mutation. In: Proceedings of 15th International Conference on Software Engineering, pp. 100–107 (1993)

  20. Ghiduk, A.S.: Reducing the number of higher-order mutants with the aid of data flow e-informatica. Softw. Eng. J. 10, 31–49 (2016)

    Google Scholar 

  21. Ma, Y.-S.; Offutt, A.J.; Kwon, Y.-R.: MuJava: an automated class mutation system. Softw. Test. Verif. Reliab. 15(2), 97–133 (2005)

    Article  Google Scholar 

  22. Jia, Y.; Harman, M.: MILU: a customizable, runtime-optimized higher order mutation testing tool for the full C language. In: Proceedings of the 3rd Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART’08), Windsor, UK: IEEE Computer Society, pp. 94–98 (2008)

  23. MuClipse. Internet http://muclipse.sourceforge.net (2008)

  24. Kintis, M.; Papadakis, M.; Malevris, N.: Isolating first order equivalent mutants via second order mutation. In: IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 701–710 (2012)

  25. Ghiduk, A.S.: Using evolutionary algorithms for higher-order mutation testing. IJCSI Int. J. Comput. Sci. 11(2), 93–104 (2014)

    Google Scholar 

  26. Allen, F.E.; Cocke, J.: A program data flow analysis procedure. Commun. ACM 19(3), 137 (1976) [Online]. https://doi.org/10.1145/360018.360025

    Article  Google Scholar 

  27. Ghiduk, A.S.; Girgis, M.R.; Hashim, M.: Higher-order mutation testing: a systematic literature review. Int. J. Comput. Sci. Rev. 25, 29–48 (2017)

    Article  MathSciNet  Google Scholar 

  28. Papadakis, M.; Henard, C.; Harman, M.; Jia, Y.; Traon, Y.L.: Threats to the validity of mutation-based test assessment. In: Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016), pp. 354–365 (2016)

  29. Umar, M.: An Evaluation of Mutation Operators for Equivalent Mutants. MS Project, Kings College, London (2006)

  30. Harman, M.; Jia, Y.; Langdon, W.B.: Strong higher order mutation-based test data generation. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11), pp. 212–222 (2011)

  31. Chekam, TT.; Papadakis, M.; Traon, YL.; Harman, M.: An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In: Proceedings of the 39th International Conference on Software Engineering (ICSE ’17), pp. 597–608 (2017)

Download references

Author information

Authors and Affiliations

  1. College of Computers and IT, Taif University, Taif, Saudi Arabia

    Ahmed S. Ghiduk & Marwa H. Shehata

  2. Department of Mathematics and Computer Science, Faculty of Science, Beni-Suef University, Beni-Suef, Egypt

    Ahmed S. Ghiduk

  3. Department of Computer Science, Faculty of Science, Minia University, Minya, Egypt

    Moheb R. Girgis

Authors
  1. Ahmed S. Ghiduk
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Moheb R. Girgis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marwa H. Shehata
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmed S. Ghiduk.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghiduk, A.S., Girgis, M.R. & Shehata, M.H. Reducing the Cost of Higher-Order Mutation Testing. Arab J Sci Eng 43, 7473–7486 (2018). https://doi.org/10.1007/s13369-018-3108-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-018-3108-3

Keywords

Search

Navigation