Skip to main content
SpringerLink
Log in

Fault detection effectiveness of mutation and data flow testing

  • Papers
  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

We report results from an experiment to compare the fault detection effectiveness of mutation, its variants and the all-uses data flow criteria. Adequate test sets were generated randomly, as opposed to by human testers as in some previous studies. We view our results in the light of those from earlier studies comparing mutation with path-oriented testing strategies. We identify and discuss factors that one might consider while evaluating an adequacy criterion for use in practice. Results from our experiments strengthen a hypothesis that an adequacy criterion based on one of the two variants of mutation has superior fault detection effectiveness than that of the all-uses criterion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Acree, A.T. (1980) On mutation, PhD thesis, School of Information and Computer Science, Georgia Institute of Technology.

  • Boyer, R.S., Elspas, B. and Levitt, K.N. (1975) SELECT — a formal system for testing and debugging programs by symbolic execution. Sigplan Notices, 10(6), 234–245.

    Google Scholar 

  • Budd, T.A. (1980) Mutation analysis of program test data, PhD thesis, Yale University.

  • Choi, B.J., DeMillo, R.A., Krauser, E.W., Mathur, A.P., Martin, R.J., Offutt, A.J., Pan, H. and Spafford, E.H. (1989) The Mothra toolset, in Proceedings of the Twenty-Second Annual Hawaii International Conference on System Sciences, HI, January, 275–84.

  • Choi, B.J. and Mathur, A.P. (1993) High performance mutation testing. Journal of Systems and Software, 20(2), 135–152.

    Article  Google Scholar 

  • Clarke, L.A. (1976) A system to generate test data and symbolically execute programs. IEEE Transactions on Software Engineering, 2(3), 215–222.

    Google Scholar 

  • Clarke, L.A., Podgurski, A., Richardson, D. J. and Zeil, S.J. (1989) A formal evaluation of data flow path selection criteria. IEEE Transactions on Software Engineering, 15(11), 1318–1332.

    Article  Google Scholar 

  • DeMillo, R.A., Lipton, R.J. and Sayward, F.G. (1978) Hints on test data selection: help for the practicing programmer. IEEE Computer. 11(4), 34–41.

    Google Scholar 

  • DeMillo, R.A., Guindi, D.S., King, K.N., McCracken, W.M. and Offutt, A.J. (1988) An extended overview of the Mothra software testing environment, in Proceedings of the Second Workshop on Software Testing. Verification and Analysis, Banff, Alberta, July, pp. 142–151.

  • DeMillo, R.A., Mathur, A.P. and Wong, W.E. (1994) Critical remarks on a hierarchy of the fault-detection ability of test methods. Technical report SERC-TR-156-P, Software Engineering Research Center, Purdue University, West Lafayette, IN.

    Google Scholar 

  • De Millo, R.A. et al. (1987) The Mothra Software Testing Environment User's Manual. Technical Report SERC-TR-4-P, Software Engineering Research Center, Purdue University, West Lafayette, IN.

    Google Scholar 

  • Frankl, P.G. and Weyuker, E.J. (1993a) A formal analysis of the fault-detecting ability of testing methods. IEEE Transactions on Software Engineering, 19(3), 202–213.

    Article  Google Scholar 

  • Frankl, P.G. and Weiss, S.N. (1993b) An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Transactions on Software Engineering, 19(3), 774–787.

    Article  Google Scholar 

  • Frankl, P.G. and Weyuker, E.J. (1993c) Provable improvements on branch testing. IEEE Transactions on Software Engineering, 19(10), 962–975.

    Article  Google Scholar 

  • Hoare, C.A.R. (1961) Algorithm 65, find. Communications of the ACM, 4(7), 321.

    Article  Google Scholar 

  • Horgan, J.R. and London, S.A. (1991) Data flow coverage and the C language, in Proceedings of the Fourth Symposium on Software Testing, Analysis, and Verification, Victoria, British Columbia, October, pp. 89–97.

  • Horgan, J.R. and London, S.A. (1992) ATAC: a data flow coverage testing tool for C, in Proceedings of Symposium on Assessment of Quality Software Development Tools, New Orleans, LA, May, pp. 2–10.

  • Howden, W.E. (1976) Reliability of the path analysis testing strategy. IEEE Transactions on Software Engineering, 2(3), 208–214.

    Google Scholar 

  • Krauser, E.W., Mathur, A.P. and Rego, V.J. (1991) High performance software testing on SIMD machines. IEEE Transactions on Software Engineering, 17(5), 403–423.

    Article  Google Scholar 

  • Laski, J.W. and Korel, B. (1983) A data flow oriented program testing strategy. IEEE Transactions on Software Engineering, 9(3), 347–354.

    Google Scholar 

  • Marick, B. (1994) The Craft of Software Testing (Prentice-Hall, New York).

    Google Scholar 

  • Mathur, A.P. (1991) Performance, effectiveness, and reliability issues in software testing, in Proceedings of the Fifteenth Annual International Computer Software and Applications Conference, Tokyo, September, pp. 604–605.

  • Mathur, A.P. and Wong, W.E. (1993) Evaluation of the cost of alternate mutation testing strategies, in Proceedings of the Seventh Brazilian Symposium on Software Engineering, Rio de Janeiro, October, pp. 320–335.

  • Mathur, A.P. and Wong, W.E. (1994a) A theoretical comparison between mutation and data flow based test adequacy criteria, in Proceedings of 1994 ACM Computer Science Conference, Phoenix, AZ, March, pp. 38–45.

  • Mathur, A.P. and Wong, W.E. (1994b) An empirical comparison of data flow and mutation based test adequacy criteria. Journal of Software Testing, Verification, and Reliability, 4(1), 9–31.

    Google Scholar 

  • Offutt, A.J., Rothermel, G. and Zapf, C. (1993) An experimental evaluation of selective mutation, in Proceedings of the 15th International Conference on Software Engineering, Baltimore, May, pp. 100–107.

  • Rapps, S. and Weyuker, E.J. (1985) Selecting software test data using data flow information. IEEE Transactions on Software Engineering, 11(4), 367–375.

    Google Scholar 

  • Walsh, P.J. (1985) A measure of test case completeness, PhD thesis, The State University of New York at Binghamton.

  • Weiss, S.N. and Fleyshgakker, V.N. (1993) Improved serial algorithms for mutation analysis, in Proceedings of International Symposium on Software Testing and Analysis, Cambridge, MA, June, pp. 149–158.

  • Weyuker, E.J. and Jeng, B. (1991) Analyzing partition testing strategies. IEEE Transactions on Software Engineering, 17(7), 703–711.

    Article  Google Scholar 

  • Weyuker, E.J., Weiss, S.N. and Hamlet, R.G. (1991) Comparison of program testing strategies, in Proceedings of the Fourth Symposium on Software Testing, Analysis and Verification, Victoria, British Columbia, October, pp. 154–164.

  • Wong, W.E. (1993) On mutation and data flow, PhD thesis, Department of Computer Science, Purdue University.

Download references

Author information

Authors and Affiliations

  1. Department of Software Technology, Hughes Network Systems, 20876, Germantown, MD, USA

    W. Eric Wong

  2. Department of Computer Sciences, Purdue University, 47907, West Lafayette, IN, USA

    Aditya P. Mathur

Authors
  1. W. Eric Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aditya P. Mathur
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wong, W.E., Mathur, A.P. Fault detection effectiveness of mutation and data flow testing. Software Qual J 4, 69–83 (1995). https://doi.org/10.1007/BF00404650

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00404650

Keywords

Search

Navigation