Skip to main content
SpringerLink
Log in

Is Higher Order Mutant Harder to Kill Than First Order Mutant? An Experimental Study

  • Conference paper
  • First Online:
Intelligent Information and Database Systems (ACIIDS 2018)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10751))

Included in the following conference series:

Abstract

This paper considers the problem whether higher order mutant is harder to kill than first order mutant or not. Higher order mutation testing has been proposed to overcome the limitations of traditional mutation testing (also called first order mutation testing) such as a large number of generated mutants, limited realism, and equivalent mutants. In this paper, we perform an empirical evaluation to answer the mentioned question with regard to the ratio of number of test cases which can kill a higher order mutant to number of test cases which can kill its constituent first order mutants. Our experimental results indicate that only a half of all generated higher order mutants are harder to kill than its constituent first order mutants.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Comput. 11(4), 34–41 (1978)

    Article  Google Scholar 

  2. Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. SE-3(4), 279–290 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  3. Nguyen, Q.V., Madeyski, L.: Problems of mutation testing and higher order mutation testing. In: van Do, T., Thi, H.A.L., Nguyen, N.T. (eds.) Advanced Computational Methods for Knowledge Engineering. AISC, vol. 282, pp. 157–172. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-06569-4_12

    Chapter  Google Scholar 

  4. Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)

    Article  Google Scholar 

  5. Jia, Y., Harman, M.: Higher order mutation testing. Inf. Softw. Technol. 51, 1379–1393 (2009)

    Article  Google Scholar 

  6. Harman, M., Jia, Y., Langdon, W.B.: A manifesto for higher order mutation testing. In: Third International Conference on Software Testing, Verification, and Validation Workshops (2010)

    Google Scholar 

  7. Offutt, A.J.: Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1, 5–20 (1992)

    Article  Google Scholar 

  8. Polo, M., Piattini, M., Garcia-Rodriguez, I.: Decreasing the cost of mutation testing with second-order mutants. Softw. Test. Verif. Reliab. 19(2), 111–131 (2008)

    Article  Google Scholar 

  9. Kintis, M., Papadakis, M., Malevris, N.: Evaluating mutation testing alternatives: a collateral experiment. In: Proceedings of the 17th Asia Pacific Software Engineering Conference (APSEC) (2010)

    Google Scholar 

  10. Papadakis, M., Malevris, N.: An empirical evaluation of the first and second order mutation testing strategies. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2010, pp. 90–99. IEEE Computer Society (2010)

    Google Scholar 

  11. Madeyski, L., Orzeszyna, W., Torkar, R., Józala, M.: Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014). https://doi.org/10.1109/tse.2013.44

    Article  Google Scholar 

  12. Omar, E., Ghosh, S.: An exploratory study of higher order mutation testing in aspect-oriented programming. In: IEEE 23rd International Symposium on Software Reliability Engineering (2012)

    Google Scholar 

  13. Jia, Y., Harman, M.: Constructing subtle faults using higher order mutation testing. In: Proceedings of the Eighth International Working Conference Source Code Analysis and Manipulation (2008)

    Google Scholar 

  14. Omar, E., Ghosh, S., Whitley, D.: Constructing subtle higher order mutants for Java and AspectJ programs. In: International Symposium on Software Reliability Engineering, pp. 340–349 (2013)

    Google Scholar 

  15. Omar, E., Ghosh, S., Whitley, D.: Comparing search techniques for finding subtle higher order mutants. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 1271–1278 (2014)

    Google Scholar 

  16. Belli, F., Güler, N., Hollmann, A., Suna, G., Yıldız, E.: Model-based higher-order mutation analysis. In: Kim, T.-H., Kim, H.-K., Khan, M.K., Kiumi, A., Fang, W.-C., Ślęzak, D. (eds.) ASEA 2010. CCIS, vol. 117, pp. 164–173. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17578-7_17

    Chapter  Google Scholar 

  17. Akinde, A.O.: Using higher order mutation for reducing equivalent mutants in mutation testing. Asian J. Comput. Sci. Inf. Technol. 2(3), 13–18 (2012)

    Google Scholar 

  18. Langdon, W.B., Harman, M., Jia, Y.: Multi-objective higher order mutation testing with genetic programming. In: Proceedings of the Fourth Testing: Academic and Industrial Conference Practice and Research (2009)

    Google Scholar 

  19. Langdon, W.B., Harman, M., Jia, Y.: Efficient multi-objective higher order mutation testing with genetic programming. J. Syst. Softw. 83, 2416–2430 (2010)

    Article  Google Scholar 

  20. Nguyen, Q.V., Madeyski, L.: Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm. In: Le Thi, H.A., Nguyen, N.T., Do, T.V. (eds.) Advanced Computational Methods for Knowledge Engineering. AISC, vol. 358, pp. 391–402. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-17996-4_35

    Chapter  Google Scholar 

  21. Nguyen, Q.V., Madeyski, L.: Empirical evaluation of multi-objective optimization algorithms searching for higher order mutants. Cybern. Syst.: Int. J. (2016)

    Google Scholar 

  22. Nguyen, Q.V., Madeyski, L.: Higher order mutation testing to drive development of new test cases: an empirical comparison of three strategies. In: Nguyen, N.T., Trawiński, B., Fujita, H., Hong, T.-P. (eds.) ACIIDS 2016. LNCS (LNAI), vol. 9621, pp. 235–244. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-49381-6_23

    Chapter  Google Scholar 

  23. Nguyen, Q.V., Madeyski, L.: On the relationship between the order of mutation testing and the properties of generated higher order mutants. In: Nguyen, N.T., Trawiński, B., Fujita, H., Hong, T.-P. (eds.) ACIIDS 2016. LNCS (LNAI), vol. 9621, pp. 245–254. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-49381-6_24

    Chapter  Google Scholar 

  24. Nguyen, Q.V., Madeyski, L.: Addressing mutation testing problems by applying multi-objective optimization algorithms and higher order mutation. J. Intell. Fuzzy Syst. 32, 1173–1182 (2017). https://doi.org/10.3233/jifs-169117

    Article  Google Scholar 

  25. Madeyski, L.: On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests. In: Münch, J., Abrahamsson, P. (eds.) PROFES 2007. LNCS, vol. 4589, pp. 207–221. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73460-4_20

    Chapter  Google Scholar 

  26. Madeyski, L.: The impact of pair programming on thoroughness and fault detection effectiveness of unit tests suites. Softw. Process: Improv. Pract. 13(3), 281–295 (2008). https://doi.org/10.1002/spip.382

    Article  Google Scholar 

  27. Madeyski, L.: The impact of test-first programming on branch coverage and mutation score indicator of unit tests: an experiment. Inf. Softw. Technol. 52(2), 169–184 (2010). https://doi.org/10.1016/j.infsof.2009.08.007

    Article  Google Scholar 

  28. Madeyski, L., Radyk, N.: Judy - a mutation testing tool for Java. IET Softw. 4(1), 32–42 (2010). https://doi.org/10.1049/iet-sen.2008.0038

    Article  Google Scholar 

  29. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multi objective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)

    Article  Google Scholar 

  30. Deb, K., Jain, H.: An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part I: solving problems with box constraints. IEEE Trans. Evol. Comput. 18(4), 577–601 (2014)

    Article  Google Scholar 

  31. Kollat, J.B., Reed, P.M.: The value of online adaptive search: a performance comparison of NSGAII, ε-NSGAII and εMOEA. In: Coello Coello, C.A., Hernández Aguirre, A., Zitzler, E. (eds.) EMO 2005. LNCS, vol. 3410, pp. 386–398. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-31880-4_27

    Chapter  Google Scholar 

  32. Deb, K., Mohan, M., Mishra, S.: A fast multi-objective evolutionary algorithm for finding well-spread pareto-optimal solutions. KenGAL, Report No. 2003002. Indian Institute of Technology, Kanpur, India (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Korea-Vietnam Friendship Information Technology College, Da Nang, Vietnam

    Quang-Vu Nguyen

  2. University of Education – Danang University, Da Nang, Vietnam

    Duong-Thu-Hang Pham

Authors
  1. Quang-Vu Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Duong-Thu-Hang Pham
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Quang-Vu Nguyen .

Editor information

Editors and Affiliations

  1. Wrocław University of Science and Technology, Wrocław, Poland

    Ngoc Thanh Nguyen

  2. Quang Binh University, Dong Hoi City, Vietnam

    Duong Hung Hoang

  3. National University of Kaohsiung, Kaohsiung, Taiwan

    Tzung-Pei Hong

  4. Rutgers University, Piscataway, New Jersey, USA

    Hoang Pham

  5. Wrocław University of Science and Technology, Wrocław, Poland

    Bogdan Trawiński

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nguyen, QV., Pham, DTH. (2018). Is Higher Order Mutant Harder to Kill Than First Order Mutant? An Experimental Study. In: Nguyen, N., Hoang, D., Hong, TP., Pham, H., Trawiński, B. (eds) Intelligent Information and Database Systems. ACIIDS 2018. Lecture Notes in Computer Science(), vol 10751. Springer, Cham. https://doi.org/10.1007/978-3-319-75417-8_62

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-75417-8_62

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-75416-1

  • Online ISBN: 978-3-319-75417-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation