Empirical Software Engineering

, Volume 14, Issue 3, pp 341–369 | Cite as

On guiding the augmentation of an automated test suite via mutation analysis

  • Ben H. Smith
  • Laurie Williams


Mutation testing has traditionally been used as a defect injection technique to assess the effectiveness of a test suite as represented by a “mutation score.” Recently, mutation testing tools have become more efficient, and industrial usage of mutation analysis is experiencing growth. Mutation analysis entails adding or modifying test cases until the test suite is sufficient to detect as many mutants as possible and the mutation score is satisfactory. The augmented test suite resulting from mutation analysis may reveal latent faults and provides a stronger test suite to detect future errors which might be injected. Software engineers often look for guidance on how to augment their test suite using information provided by line and/or branch coverage tools. As the use of mutation analysis grows, software engineers will want to know how the emerging technique compares with and/or complements coverage analysis for guiding the augmentation of an automated test suite. Additionally, software engineers can benefit from an enhanced understanding of efficient mutation analysis techniques. To address these needs for additional information about mutation analysis, we conducted an empirical study of the use of mutation analysis on two open source projects. Our results indicate that a focused effort on increasing mutation score leads to a corresponding increase in line and branch coverage to the point that line coverage, branch coverage and mutation score reach a maximum but leave some types of code structures uncovered. Mutation analysis guides the creation of additional “common programmer error” tests beyond those written to increase line and branch coverage. We also found that 74% of our chosen set of mutation operators is useful, on average, for producing new tests. The remaining 26% of mutation operators did not produce new test cases because their mutants were immediately detected by the initial test suite, indirectly detected by test suites we added to detect other mutants, or were not able to be detected by any test.


Mutation testing Line coverage Fault injection Empirical effectiveness Test case augmentation Mutation analysis Mutation testing tool Statement coverage Test adequacy Web application Open source Unit testing 



We would like to thank the North Carolina State University RealSearch group for their helpful comments on the paper. Funding for this research was provided in part by an IBM Faculty Award. This work is also supported by the National Science Foundation under CAREER Grant No. 0346903. Any opinions expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.


  1. Alexander RT, Bieman JM, Ghosh S, Bixia J (2002) Mutation of Java objects. 13th International Symposium on Software Reliability Engineering, Fort Collins, CO USA, pp 341–351Google Scholar
  2. Andrews JH, Briand LC, Labiche Y (2005) Is mutation an appropriate tool for testing experiments? 27th International Conference on Software Engineering, St. Louis, MO, USA, pp 402–411Google Scholar
  3. Andrews JH, Briand LC, Labiche Y, Namin AS (2006) Using mutation analysis for assessing and comparing testing coverage criteria. Software Engineering. IEEE Trans 32(8):608–624Google Scholar
  4. DeMillo RA, Lipton RJ, Sayward FG (1978) Hints on test data selection: help for the practicing programmer. Computer 11(4):34–41. doi: 10.1109/C-M.1978.218136 CrossRefGoogle Scholar
  5. DeMillo RA, Guindi DS, McCracken WM, Offutt AJ, King KN (1988) An extended overview of the Mothra software testing environment. Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, Banff, Alta., Canada, pp 142–151Google Scholar
  6. Frankl PG, Weiss SN, Hu C (1997) All uses vs mutation testing: an experimental comparison of effectiveness. J Syst Softw 38(3):235–253. doi: 10.1016/S0164-1212(96)00154-9 CrossRefGoogle Scholar
  7. Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley Longman, Boston, MA, USAGoogle Scholar
  8. Hierons RM, Harman M, Danicic S (1999) Using program slicing to assist in the detection of equivalent mutants. Softw Test Verif Reliab 9(4):233–262CrossRefGoogle Scholar
  9. Hovemeyer D, Pugh W (2004) Finding bugs is easy. ACM SIGPLAN Not 39(12):92–106CrossRefGoogle Scholar
  10. IEEE (1990) IEEE Standard 610.12–1990, IEEE Standard Glossary of Software Engineering TerminologyGoogle Scholar
  11. Irvine SA, Pavlinic T, Trigg L, Cleary JG, Inglis S, Utting M (2007) Jumble Java byte code to measure the effectiveness of unit tests. Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION, 2007. TAICPART-MUTATION 2007, pp 169–175Google Scholar
  12. Kaner C (1996) Software negligence and testing coverage. Proceedings of STAR, Orlando, FL, pp 1–13Google Scholar
  13. Krishnamurthy S, Mathur A (1996) On predicting reliability of modules using code coverage. Proceedings of the 1996 conference of the Centre for Advanced Studies on Collaborative research, Toronto, Ontario, Canada, pp 1–12Google Scholar
  14. Ma YS, Offut J (2005a) Description of class mutation operators for Java,, accessed 4/19/2007
  15. Ma YS, Offut J (2005b) Description of method-level mutation operators for Java,, accessed 4/19/2007
  16. Ma YS, Harrold MJ, Kwon YR (2006) Evaluation of mutation testing for object-oriented programs. 28th International Conference on Software Engineering, Shanghai, China, pp 869–872Google Scholar
  17. Murnane T, Reed K, Assoc T, Carlton V (2001) On the effectiveness of mutation analysis as a black box testing technique. Software Engineering Conference, Canberra, ACT, Australia, pp 12–20Google Scholar
  18. Namin AS, Andrews JH (2006) Finding sufficient mutation operators via variable reduction. Mutation Analysis, 2006. Second Workshop on, pp 5–5Google Scholar
  19. Offutt J, Lee A, Rothermel G, Untch RH, Zapf C (1996a) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol 5(2):99–118CrossRefGoogle Scholar
  20. Offutt J, Pan J, Tewary K, Zhang T (1996b) An experimental evaluation of data flow and mutation testing. Softw Pract Exp 26(2):165–176CrossRefGoogle Scholar
  21. Offutt J, Ma YS, Kwon YR (2004) An experimental mutation system for Java. ACM SIGSOFT Softw Eng Notes 29(5):1–4Google Scholar
  22. Offutt J, Ma YS, Kwon YR (2006) The class-level mutants of MuJava. International Workshop on Automation of Software Testing, Shanghai, China, pp 78–84Google Scholar
  23. Smith BH, Williams L (2007) An empirical evaluation of the MuJava mutation operators. Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION, 2007, Windsor, UK, pp 193–202Google Scholar
  24. Voas J, Miller KW (1995) Using fault-injection to assess software engineering standards. Second IEEE International Software Engineering Standards Symposium ‘Experience and Practice’, Montreal, Que., Canada, pp 318–336Google Scholar
  25. Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4):366–427CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  1. 1.Department of Computer ScienceNorth Carolina State UniversityRaleighUSA

Personalised recommendations