Skip to main content
SpringerLink
Log in

Is Mutation Testing Ready to Be Adopted Industry-Wide?

  • Conference paper
  • First Online:
Product-Focused Software Process Improvement (PROFES 2016)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10027))

Abstract

Mutation Testing has a long research history as a way to improve the quality of software tests. However, it has not yet reached wide consensus for industry-wide adoption, mainly due to missing clear benefits and computational complexity for the application to large systems. In this paper, we investigate the current state of mutation testing support for Java Virtual Machine (JVM) environments. By running an experimental evaluation, we found out that while default configurations are unbearable for larger projects, using strategies such as selective operators, second order mutation and multi-threading can increase the applicability of the approach. However, there is a trade-off in terms of quality of the achieved results of the mutation analysis process that needs to be taken into account.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://gist.github.com/netj/526585.

  2. 2.

    the experimental package is available at https://goo.gl/5GPdQv.

  3. 3.

    description of operators can be found at http://pitest.org/quickstart/mutators/ and http://mutationtesting.org/judy/documentation/.

References

  1. CJBS Insight: Cambridge university study states software bugs cost economy $312 billion per year. http://insight.jbs.cam.ac.uk/2013/financial-content-cambridge-university-study-states-software-bugs-cost-economy-312-billion-per-year/

  2. Crispin, L., Gregory, J.: Agile Testing: A Practical Guide for Testers and Agile Teams. Pearson Education, Boston (2009)

    Google Scholar 

  3. Dedík, V., Rossi, B.: Automated bug triaging in an industrial context. In: 42nd EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 363–367. IEEE (2016)

    Google Scholar 

  4. Delahaye, M., Du Bousquet, L.: A comparison of mutation analysis tools for Java. In: 13th International Conference on Quality Software (QSIC), pp. 187–195. IEEE (2013)

    Google Scholar 

  5. DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. Comput. 11(4), 34–41 (1978)

    Article  Google Scholar 

  6. Jia, Y., Harman, M.: Higher order mutation testing. Inf. Softw. Technol. 51(10), 1379–1393 (2009)

    Article  Google Scholar 

  7. Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)

    Article  Google Scholar 

  8. Klischies, D., Fögen, K.: An analysis of current mutation testing techniques applied to real world examples. In: Full-scale Software Engineering/Current Trends in Release Engineering, p. 13 (2016)

    Google Scholar 

  9. Madeyski, L., Orzeszyna, W., Torkar, R., Jozala, M.: Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014)

    Article  Google Scholar 

  10. Madeyski, L., Radyk, N.: Judy-a mutation testing tool for Java. IET Softw. 4(1), 32–42 (2010)

    Article  Google Scholar 

  11. Nica, S., Ramler, R., Wotawa, F.: Is mutation testing scalable for real-world software projects. In: VALID Third International Conference on Advances in System Testing and Validation Lifecycle, Barcelona, Spain (2011)

    Google Scholar 

  12. Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. 5(2), 99–118 (1996)

    Article  Google Scholar 

  13. Offutt, A.J., Rothermel, G., Zapf, C.: An experimental evaluation of selective mutation. In: Proceedings of the 15th International Conference on Software Engineering ICSE 1993, pp. 100–107. IEEE Computer Society Press, Los Alamitos (1993)

    Google Scholar 

  14. Offutt, A.J., Untch, R.H.: Mutation 2000: uniting the orthogonal. In: Wong, W.E. (ed.) Mutation Testing for the New Century, pp. 34–44. Kluwer Academic Publishers, Norwell (2001)

    Chapter  Google Scholar 

  15. Rani, S., Suri, B., Khatri, S.K.: Experimental comparison of automated mutation testing tools for Java. In: 2015 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO), pp. 1–6. IEEE (2015)

    Google Scholar 

  16. Roy, N.K.S., Rossi, B.: Towards an improvement of bug severity classification. In: 40th EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 269–276. IEEE (2014)

    Google Scholar 

  17. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Springer, Heidelberg (2012)

    Book  MATH  Google Scholar 

Download references

Acknowledgments

We are grateful to the developers of both PITest and Judy for feedback provided in the usage of the tools. In case of Judy, the SOM experiments have been possible with a newer version provided by the developers.

Author information

Authors and Affiliations

  1. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Jakub Možucha & Bruno Rossi

Authors
  1. Jakub Možucha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno Rossi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bruno Rossi .

Editor information

Editors and Affiliations

  1. Norwegian University of Science and Technology, Trondheim, Norway

    Pekka Abrahamsson

  2. Fraunhofer Institute for Experimental Software Engineering, Kaiserslautern, Germany

    Andreas Jedlitschka

  3. and Technology, Norwegian University of Science and Technology, Trondheim, Norway

    Anh Nguyen Duc

  4. University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  5. Okayama Prefectural University, Soja, Japan

    Sousuke Amasaki

  6. Tampere University of Technology, Tampere, Finland

    Tommi Mikkonen

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Možucha, J., Rossi, B. (2016). Is Mutation Testing Ready to Be Adopted Industry-Wide?. In: Abrahamsson, P., Jedlitschka, A., Nguyen Duc, A., Felderer, M., Amasaki, S., Mikkonen, T. (eds) Product-Focused Software Process Improvement. PROFES 2016. Lecture Notes in Computer Science(), vol 10027. Springer, Cham. https://doi.org/10.1007/978-3-319-49094-6_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-49094-6_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-49093-9

  • Online ISBN: 978-3-319-49094-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation