Problems of Mutation Testing and Higher Order Mutation Testing

Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 282)

Abstract

Since Mutation Testing was proposed in the 1970s, it has been considered as an effective technique of software testing process for evaluating the quality of the test data. In other words, Mutation Testing is used to evaluate the fault detection capability of the test data by inserting errors into the original program to generate mutations, and after then check whether tests are good enough to detect them. However, the problems of mutation testing such as a large number of generated mutants or the existence of equivalent mutants, are really big barriers for applying mutation testing. A lot of solutions have been proposed to solve that problems. A new form of Mutation Testing, Higher Order Mutation Testing, was first proposed by Harman and Jia in 2009 and is one of the most promising solutions. In this paper, we consider the main limitations of Mutation Testing and previous proposed solutions to solve that problems. This paper also refers to the development of Higher Order Mutation Testing and reviews the methods for finding the good Higher Order Mutants.

Keywords

Mutation Testing Higher Order Mutation Higher Order Mutants 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Computer 11(4), 34–41 (1978)CrossRefGoogle Scholar
  2. 2.
    Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering SE-3(4), 279–290 (1977)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Jia, Y., Harman, M.: Higher order mutation testing. Information and Software Technology 51, 1379–1393 (2009)CrossRefGoogle Scholar
  4. 4.
    Harman, M., Jia, Y., Langdon, W.B.: A Manifesto for Higher Order Mutation Testing. In: Third International Conf. on Software Testing, Verification, and Validation Workshops (2010)Google Scholar
  5. 5.
    Afzal, W., Torkar, R., Feldt, R.: A systematic review of search-based testing for non-functional system properties. Information and Software Technology 51(6), 957–976 (2009)CrossRefGoogle Scholar
  6. 6.
    Harman, M.: The current state and future of search based software engineering. In: Briand, L., Wolf, A. (eds.) Future of Software Engineering 2007, pp. 342–357. IEEE Computer Society Press, Los Alamitos (2007)Google Scholar
  7. 7.
    Harman, M., Mansouri, A., Zhang, Y.: Search based software engineering: A comprehensive analysis and review of trends techniques and applications. Technical Report TR-09-03, Department of Computer Science, King’s College London (2009)Google Scholar
  8. 8.
    Raiha, O.: A survey on search based software design. Technical Report Technical Report D-2009-1, Department of Computer Sciences, University of Tamper (2009)Google Scholar
  9. 9.
    Langdon, W.B., Harman, M., Jia, Y.: Efficient multi-objective higher order mutation testing with genetic programming. The Journal of Systems and Software 83 (2010)Google Scholar
  10. 10.
    Jia, Y., Harman, M.: Constructing Subtle Faults Using Higher Order Mutation Testing. In: Proc. Eighth Int’l Working Conf. Source Code Analysis and Manipulation (2008)Google Scholar
  11. 11.
    Jia, Y., Harman, M.: MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language. In: Proceedings of the 3rd Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART 2008), pp. 94–98. IEEE Computer Society, Windsor (2008)CrossRefGoogle Scholar
  12. 12.
    Offutt, A.J.: Investigations of the software testing coupling effect. ACM Transactions on Software Engineering and Methodology 1(1), 5–20 (1992)CrossRefGoogle Scholar
  13. 13.
    Adamopoulos, K., Harman, M., Hierons, R.M.: How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution. In: Deb, K., Tari, Z. (eds.) GECCO 2004. LNCS, vol. 3103, pp. 1338–1349. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  14. 14.
    Acree, A.T.: On Mutation. PhD thesis, Georgia Inst. of Technology (1980)Google Scholar
  15. 15.
    Budd, T.A.: Mutation Analysis of Program Test Data. PhD thesis, Yale Univ. (1980)Google Scholar
  16. 16.
    DeMillo, R.A., Guindi, D.S., King, K.N., McCracken, W.M., Offutt, A.J.: An Extended Overview of the Mothra Software Testing Environment. In: Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, pp. 142–151 (1988)Google Scholar
  17. 17.
    Sahinoglu, M., Spafford, E.H.: A Bayes Sequential Statistical Procedure for Approving Software Products. In: Proc. IFIP Conf. Approving Software Products, pp. 43–56 (1990)Google Scholar
  18. 18.
    King, K.N., Offutt, A.J.: A Fortran Language System for Mutation-Based Software Testing. Software: Practice and Experience 21(7), 685–718 (1991)Google Scholar
  19. 19.
    Mathur, A.P., Wong, W.E.: An Empirical Comparison of Mutation and Data Flow Based Test Adequacy Criteria. Technical Report, Purdue Univ. (1993)Google Scholar
  20. 20.
    Wong, W.E.: On Mutation and Data Flow. PhD thesis, Purdue Univ. (1993)Google Scholar
  21. 21.
    Hussain, S.: Mutation Clustering. Master’s thesis, King’s College London (2008)Google Scholar
  22. 22.
    Ji, C., Chen, Z., Xu, B., Zhao, Z.: A Novel Method of Mutation Clustering Based on Domain Analysis. In: Proc. 21st Int’l Conf. Software Eng. and Knowledge Eng. (2009)Google Scholar
  23. 23.
    Agrawal, H., DeMillo, R.A., Hathaway, B., Hsu, W., Krauser, E.W., Martin, R.J., Mathur, A.P., Spafford, E.: Design of Mutant Operators for the C Programming Language. Technical Report SERC-TR-41-P, Purdue Univ. (1989)Google Scholar
  24. 24.
    Mathur, A.P.: Performance, Effectiveness, and Reliability Issues in Software Testing. In: Proc. Fifth Int’l Computer Software and Applications Conf., pp. 604–605 (1991)Google Scholar
  25. 25.
    Offutt, A.J., Rothermel, G., Zapf, C.: An Experimental Evaluation of Selective Mutation. In: Proc. 15th Int’l Conf. Software Eng., pp. 100–107 (1993)Google Scholar
  26. 26.
    Wong, W.E., Mathur, A.P.: Reducing the Cost of Mutation Testing: An Empirical Study. J. Systems and Software 31(3), 185–196 (1995)CrossRefGoogle Scholar
  27. 27.
    Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An Experimental Determination of Sufficient Mutant Operators. ACM Trans. Soft. Eng. and Methodology (1996)Google Scholar
  28. 28.
    Mresa, E.S., Bottaci, L.: Efficiency of Mutation Operators and Selective Mutation Strategies: An Empirical Study. Software Testing, Verification, and Reliability 9(4), 205–232 (1999)CrossRefGoogle Scholar
  29. 29.
    Barbosa, E.F., Maldonado, J.C., Vincenzi, A.M.R.: Toward the Determination of Sufficient Mutant Operators for C. Software Testing, Verification, and Reliability 11(2), 113–136 (2001)CrossRefGoogle Scholar
  30. 30.
    Namin, A.S., Andrews, J.H.: Finding Sufficient Mutation Operators via Variable Reduction. In: Proc. Second Workshop Mutation Analysis, p. 5 (2006)Google Scholar
  31. 31.
    Namin, A.S., Andrews, J.H.: On Sufficiency of Mutants. In: Proc. 29th Int’l Conf. Software Eng., pp. 73–74 (2007)Google Scholar
  32. 32.
    Namin, A.S., Andrews, J.H., Murdoch, D.J.: Sufficient Mutation Operators for Measuring Test Effectiveness. In: Proc. 30th Int’l Conf. Software Eng., pp. 351–360 (2008)Google Scholar
  33. 33.
    Polo, M., Piattini, M., Garcia-Rodriguez, I.: Decreasing the Cost of Mutation Testing with Second-Order Mutants. Software Testing, Verification, and Reliability 19(2), 111–131 (2008)CrossRefGoogle Scholar
  34. 34.
    Purushothaman, R., Perry, D.E.: Toward Understanding the Rhetoric of small source code changes. IEEE Transactions on Software Engineering 31(6) (2005)Google Scholar
  35. 35.
    Langdon, W.B., Harman, M., Jia, Y.: Multi Objective Higher Order Mutation Testing with Genetic Programming. In: Proc. Fourth Testing: Academic and Industrial Conf. Practice and Research (2009) Google Scholar
  36. 36.
    Howden, W.E.: Weak Mutation Testing and Completeness of Test Sets. IEEE Trans. Soft. Eng. 8(4), 371–379 (1982)CrossRefGoogle Scholar
  37. 37.
    Girgis, M.R., Woodward, M.R.: An Integrated System for Program Testing Using WeakMutation and Data Flow Analysis. In: Proc. Eighth Int’l Conf. Software Eng. (1985)Google Scholar
  38. 38.
    Horgan, J.R., Mathur, A.P.: Weak Mutation is Probably Strong Mutation. Technical Report SERC-TR-83-P, Purdue Univ. (1990)Google Scholar
  39. 39.
    Woodward, M.R.: Mutation Testing-An Evolving Technique. In: Proc. IEE Colloquium on Software Testing for Critical Systems, pp. 3/1–3/6 (1990)Google Scholar
  40. 40.
    Marick, B.: The Weak Mutation Hypothesis. In: Proc. Fourth Symp. Software Testing, Analysis and Verification, pp. 190–199 (1991)Google Scholar
  41. 41.
    Offutt, A.J., Lee, S.D.: How Strong is Weak Mutation. In: Proc. Fourth Symp. Software Testing, Analysis and Verification, pp. 200–213 (1991)Google Scholar
  42. 42.
    Offutt, A.J., Lee, S.D.: An Empirical Evaluation of Weak Mutation. IEEE Trans. Software Eng. 20(5), 337–344 (1994)CrossRefGoogle Scholar
  43. 43.
    Woodward, M.R., Halewood, K.: From Weak to Strong Dead or Alive? An Analysis of Some Mutationtesting Issues. In: Proc. Second Workshop on Software Testing, Verification, and Analysis, pp. 152–158 (1988)Google Scholar
  44. 44.
    Jackson, D., Woodward, M.R.: Parallel Firm Mutation of Java Programs. In: Proc. First Workshop on Mutation Analysis, pp. 55–61 (2000)Google Scholar
  45. 45.
    Offutt, A.J., King, K.N.: A Fortran 77 Interpreter for Mutation Analysis. ACM SIGPLAN Notices 22(7), 177–188 (1987)CrossRefGoogle Scholar
  46. 46.
    Choi, B., Mathur, A.P.: High-Performance Mutation Testing. J. Systems and Software 20(2), 135–152 (1993)CrossRefGoogle Scholar
  47. 47.
    Delamaro, M.E.: Proteum-A Mutation Analysis Based Testing Environment. Master’s thesis, Univ. of Sao Paulo (1993)Google Scholar
  48. 48.
    Delamaro, M.E., Maldonado, J.C.: Proteum: A Tool for the Assessment of Test Adequacy for C Programs. In: Proc. Conf. Performability in Computing Systems (1996)Google Scholar
  49. 49.
    DeMillo, R.A., Krauser, E.W., Mathur, A.P.: Compiler-Integrated Program Mutation. In: Proc. Fifth Ann. Computer Software and Applications Conf., pp. 351–356 (1991)Google Scholar
  50. 50.
    Krauser, E.W.: Compiler-Integrated Software Testing. PhD thesis, Purdue Univ. (1991)Google Scholar
  51. 51.
    Untch, R.H.: Mutation-Based Software Testing Using Program Schemata. In: Proc. 30th Ann. Southeast Regional Conf., pp. 285–291 (1992)Google Scholar
  52. 52.
    Mathur, A.P.: CS 406 Software Engineering I. Course Project Handout (1992)Google Scholar
  53. 53.
    Untch, R.H., Offutt, A.J., Harrold, M.J.: Mutation Analysis Using Mutant Schemata. In: Proc. Int’l Symp. Software Testing and Analysis, pp. 139–148 (1993)Google Scholar
  54. 54.
    Untch, R.H.: Schema-Based Mutation Analysis: A New Test Data Adequacy Assessment Method. PhD thesis, Clemson Univ. (1995)Google Scholar
  55. 55.
    Offutt, A.J., Ma, Y.S., Kwon, Y.R.: An Experimental Mutation System for Java. ACM SIGSOFT Software Eng. Notes 29(5), 1–4 (2004)Google Scholar
  56. 56.
    Ma, Y.S., Offutt, A.J., Kwon, Y.R.: MuJava: An Automated Class Mutation System. Software Testing, Verification, and Reliability 15(2), 97–133 (2005)CrossRefGoogle Scholar
  57. 57.
    Ma, Y.S., Offutt, A.J., Kwon, Y.R.: MuJava: A Mutation System for Java. In: Proc. 28th Int’l Conf. Software Eng., pp. 827–830 (2006)Google Scholar
  58. 58.
    Schuler, D., Dallmeier, V., Zeller, A.: Efficient Mutation Testing by Checking Invariant Violations. In: Proc. Int’l Symp. Software Testing and Analysis (2009)Google Scholar
  59. 59.
    Bogacki, B., Walter, B.: Evaluation of Test Code Quality with Aspect-Oriented Mutations. In: Abrahamsson, P., Marchesi, M., Succi, G. (eds.) XP 2006. LNCS, vol. 4044, pp. 202–204. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  60. 60.
    Bogacki, B., Walter, B.: Aspect-Oriented Response Injection: An Alternative to Classical Mutation Testing. In: Sacha, K. (ed.) Soft. Eng. Techniques: Design for Quality. IFIP, pp. 273–282. Springer, Boston (2007)Google Scholar
  61. 61.
    Mathur, A.P., Krauser, E.W.: Mutant Unification for Improved Vectorization. Technical Report SERC-TR-14-P, Purdue Univ. (1988)Google Scholar
  62. 62.
    Krauser, E.W., Mathur, A.P., Rego, V.J.: High Performance Software Testing on SIMD Machines. In: Proc. Second Workshop on Software Testing, Verification and Analysis (1988)Google Scholar
  63. 63.
    Krauser, E.W., Mathur, A.P., Rego, V.J.: High Performance Software Testing on SIMD Machines. IEEE Trans. Software Eng. 17(5), 403–423 (1991)CrossRefGoogle Scholar
  64. 64.
    Offutt, A.J., Pargas, R.P., Fichter, S.V., Khambekar, P.K.: Mutation Testing of Software Using a MIMD Computer. In: Proc. Int’l Conf. Parallel Processing, pp. 255–266 (1992)Google Scholar
  65. 65.
    Zapf, C.N.: A Distributed Interpreter for the Mothra Mutation Testing System. Master’s thesis, Clemson Univ. (1993)Google Scholar
  66. 66.
    Weiss, S.N., Fleyshgakker, V.N.: Improved Serial Algorithms for Mutation Analysis. ACM SIGSOFT Software Eng. Notes 18(3), 149–158 (1993)CrossRefGoogle Scholar
  67. 67.
    Fleyshgakker, V.N., Weiss, S.N.: Efficient Mutation Analysis: A New Approach. In: Proc. Int’l Symp. Software Testing and Analysis, pp. 185–195 (1994)Google Scholar
  68. 68.
    Madeyski, L., Orzeszyna, W., Torkar, R., Józala, M.: Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering (2013), http://dx.doi.org/10.1109/TSE.2013.44 (accepted)
  69. 69.
    Baldwin, D., Sayward, F.G.: Heuristics for determin-ing equivalence of program mutations. Yale University, New Haven, Connecticut, Tech. Report 276 (1979)Google Scholar
  70. 70.
    Offutt, A.J., Craft, W.M.: Using compiler optimization techniques to detect equivalent mutants. Software Testing, Verification and Reliability 4(3), 131–154 (1994)CrossRefGoogle Scholar
  71. 71.
    Offutt, A.J., Pan, J.: Detecting equivalent mutants and the feasible path problem. In: Proc. Eleventh Annual Conf. ‘Systems Integrity Computer Assurance COMPASS 1996 Software Safety. Process Security’, pp. 224–236 (1996)Google Scholar
  72. 72.
    Hierons, R., Harman, M., Danicic, S.: Using program slicing to assist in the detection of equivalent mutants. Software Testing, Verification and Reliability (1999)Google Scholar
  73. 73.
    Ellims, M., Ince, D., Petre, M.: The Csaw C mutation tool: Initial results. In: Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques- MUTATION, pp. 185–192. IEEE Computer Society Press, Washington, DC (2007)Google Scholar
  74. 74.
    Martin, E., Xie, T.: A fault model and mutation testing of access control policies. In: Proceedings of the 16th International Conference on World Wide Web. WWW 2007, pp. 667–676. ACM Press, New York (2007)CrossRefGoogle Scholar
  75. 75.
    DuBousquet, L., Delaunay, M.: Towards mutation analysis for Lustre programs. Electronic Notes in Theoretical Computer Science 203(4), 35–48 (2008)CrossRefGoogle Scholar
  76. 76.
    Offutt, A.J.: Investigations of the software testing coupling effect. ACM Transactions on Software Engineering Methodology 1, 5–20 (1992)CrossRefGoogle Scholar
  77. 77.
    Mresa, E.S., Bottaci, L.: Efficiency of mutation operators and selective mutation strategies: An empirical study. Software Testing, Verification and Reliability (1999)Google Scholar
  78. 78.
    Harman, M., Hierons, R., Danicic, S.: The relationship between program dependence and mutation analysis. In: Wong, W.E. (ed.) Mutation Testing for the New Century, pp. 5–13. Kluwer Academic Publishers, Norwell (2001)CrossRefGoogle Scholar
  79. 79.
    Adamopoulos, K., Harman, M., Hierons, R.M.: How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In: Deb, K., Tari, Z. (eds.) GECCO 2004. LNCS, vol. 3103, pp. 1338–1349. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  80. 80.
    Offutt, A.J., Ma, Y.S., Kwon, Y.R.: The class-level mutants of MuJava. In: Proceedings of the 2006 International Workshop on Automation of Software Test - AST 2006. AST 2006, pp. 78–84. ACM Press, New York (2006)CrossRefGoogle Scholar
  81. 81.
    Kaminski, G., Ammann, P.: Using a fault hierarchy to improve the efficiency of DNF logic mutation testing. In: Proc. Int. Conf. Software Testing Verification and Validation, ICST 2009, pp. 386–395 (2009)Google Scholar
  82. 82.
    Ji, C., Chen, Z., Xu, B., Wang, Z.: A new mutation analysis method for testing Java exception handling. In: Proc. 33rd Annual IEEE Int. Computer Software and Applications Conf., COMPSAC 2009, vol. 2, pp. 556–561 (2009)Google Scholar
  83. 83.
    Kintis, M., Papadakis, M., Malevris, N.: Evaluating mutation testing alternatives: A collateral experiment. In: Proc. 17th Asia Pacific Soft. Eng. Conf. (APSEC) (2010)Google Scholar
  84. 84.
    Papadakis, M., Malevris, N.: An empirical evaluation of the first and second order mutation testing strategies. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops. ICSTW 2010, pp. 90–99. IEEE Computer Society (2010)Google Scholar
  85. 85.
    Vincenzi, A.M.R., Nakagawa, E.Y., Maldonado, J.C., Delamaro, M.E., Romero, R.A.F.: Bayesian-learning based guide-lines to determine equivalent mutants. International Journal of Soft. Eng. and Knowledge Engineering 12(6), 675–690 (2002)CrossRefGoogle Scholar
  86. 86.
    Grün, B.J.M., Schuler, D., Zeller, A.: The impact of equivalent mutants. In: Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation Workshops, pp. 192–199. IEEE Computer Society, Denver (2009)Google Scholar
  87. 87.
    Schuler, D., Dallmeier, V., Zeller, A.: Efficient mutation testing by checking invariant violations. In: Proceedings of the Eighteenth International Symposium on Software Testing and Analysis. ISSTA 2009. ACM Press, New York (2009)Google Scholar
  88. 88.
    Schuler, D., Zeller, A.: (Un-)covering equivalent mutants. In: Proceedings of the 3rd International Conference on Software Testing Verification and Validation (ICST 2010), Paris, France, pp. 45–54 (2010)Google Scholar
  89. 89.
    Schuler, D., Zeller, A.: Covering and uncovering equivalent mutants. Software Testing, Verification and Reliability 23(5), 353–374 (2012)CrossRefGoogle Scholar
  90. 90.
    Offutt, A.J., Pan, J.: Automatically Detecting Equivalent Mutants and Infeasible Paths. Software Testing, Verification and Reliability 7(3), 165–192 (1997)CrossRefGoogle Scholar
  91. 91.
    Clark, J.A., Dan, H., Hierons, R.M.: Semantic Mutation Testing. In: Third International Conf. on Software Testing, Verification and Validation Workshops (2010)Google Scholar
  92. 92.
    Dan, H., Hierons, R.M.: Semantic Mutation Analysis of Floating-point Comparison. In: IEEE Fifth International Conference on Software Testing, Verification and Validation (2012)Google Scholar
  93. 93.
    Boonyakulsrirung, P., Suwannasare, T.: A Weak Mutation Testing Framework for WS-BPEL. In: Eighth International Joint Conf. on Computer Science and Soft. Engineering (2011)Google Scholar
  94. 94.
    Durelli, V.H.S., Offutt, A.J., Delamaro, M.E.: Toward Harnessing High-level Language Virtual Machines for Further Speeding up Weak Mutation Testing. In: IEEE Fifth International Conference on Software Testing, Verification and Validation (2012)Google Scholar
  95. 95.
    Mateo, P.R., Usaola, M.P.: Mutant Execution Cost Reduction Through MUSIC (MUtant Schema Improved with extra Code). In: IEEE Fifth International Conference on Software Testing, Verification and Validation (2012)Google Scholar
  96. 96.
    Lisherness, P., Lesperance, N., Cheng, K.T.: Mutation Analysis with Coverage Discounting. In: Design, Automation and Test in Europe Conference and Exhibition (2013)Google Scholar
  97. 97.
    Polo, M., Piattini, M., Garcia-Rodriguez, I.: Decreasing the Cost of Mutation Testing with Second-Order Mutants. Software Testing, Verification, and Reliability 19(2), 111–131 (2008)CrossRefGoogle Scholar
  98. 98.
    Madeyski, L., Radyk, N.: Judy - a mutation testing tool for Java. IET Software 4(1), 32–42 (2010), http://madeyski.e-informatyka.pl/download/Madeyski10b.pdf CrossRefGoogle Scholar
  99. 99.
    Madeyski, L.: On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests. In: Münch, J., Abrahamsson, P. (eds.) PROFES 2007. LNCS, vol. 4589, pp. 207–221. Springer, Heidelberg (2007), http://madeyski.e-informatyka.pl/download/Madeyski07.pdf CrossRefGoogle Scholar
  100. 100.
    Madeyski, L.: Impact of pair programming on thoroughness and fault detection effectiveness of unit test suites. Software Process: Improvement and Practice 13(3), 281–295 (2008), http://madeyski.e-informatyka.pl/download/Madeyski08.pdf CrossRefGoogle Scholar
  101. 101.
    Madeyski, L.: The impact of test-first programming on branch coverage and mutation score indicator of unit tests: An experiment. Information and Software Technology 52, 169–184 (2010), http://madeyski.e-informatyka.pl/download/Madeyski10c.pdf CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  1. 1.Institute of InformaticsWroclaw University of TechnologyWroclawPoland

Personalised recommendations