Advertisement

An approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing

  • Atul Gupta
  • Pankaj Jalote
TACAS 06

Abstract

Experimental work in software testing has generally focused on evaluating the effectiveness and effort requirements of various coverage criteria. The important issue of testing efficiency has not been sufficiently addressed. In this paper, we describe an approach for comparing the effectiveness and efficiency of test coverage criteria using mutation analysis. For each coverage criterion under study, we generate multiple coverage-adequate minimal test suites for a test-program from a test-pool, which are then executed on a set of systematically generated program mutants to obtain the fault data. We demonstrate the applicability of the proposed approach by describing the results of an experiment comparing the three code-based testing criteria, namely, block coverage, branch coverage, and predicate coverage. Our results suggest that there is a trade-off between effectiveness and efficiency of a coverage criterion. Specifically, the predicate coverage criterion was found to be most effective but least efficient whereas the block coverage criterion was most efficient but least effective. We observed high variability in the performance of block test suites whereas branch and predicate test suites were relatively consistent. Overall results suggest that the branch coverage criterion performs consistently with good efficiency and effectiveness, and it appears to be the most viable option for code-based control flow testing.

Keywords

Coverage based testing Coverage criteria Test case generation Experimental evaluation Mutation operators Statistical analysis 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Andrews, J.H., Briand, L.C., Labiche, Y.: Is mutation an appropriate tool for testing experiments? In: ICSE ’05: Proceedings of the 27th International Conference on Software Engineering, pp. 402–411. ACM Press, New York (2005)Google Scholar
  2. 2.
    Antoniol, G., Briand, L.C., Penta, M.D., Labiche, Y.: A case study using the round-trip strategy for state-based class testing. In: ISSRE ’02: Proceedings of the 13th International Symposium on Software Reliability Engineering, pp. 269–279. IEEE Computer Society, Washington (2002)Google Scholar
  3. 3.
    Basili, V.R.: The role of experimentation in software engineering: past, current, and future. In: ICSE ’96: Proceedings of the 18th International Conference on Software Engineering, pp. 442–449. IEEE Computer Society, Washington (1996)Google Scholar
  4. 4.
    Basili, V.R., Caldeira, G., Rombach, H.D.: The goal question metric approach. In: Encyclopedia of Software Engineering (1994)Google Scholar
  5. 5.
    Basili V.R. and Selby R.W. (1987). Comparing the effectiveness of software testing strategies. IEEE Trans. Softw. Eng. 13(12): 1278–1296 CrossRefGoogle Scholar
  6. 6.
    Basili V.R., Selby R.W. and Hutchens D.H. (1986). Experimentation in software engineering. IEEE Trans. Softw. Eng. 12(7): 733–743 Google Scholar
  7. 7.
  8. 8.
    Briand L.C. and Labiche Y. (2004). Empirical studies of software testing techniques: challenges, practical strategies and future research. SIGSOFT Softw. Eng. Notes 29(5): 1–3 CrossRefGoogle Scholar
  9. 9.
    Briand, L.C., Labiche, Y., Wang, Y.: Using simulation to empirically investigate test coverage criteria based on statechart. In: ICSE ’04: Proceedings of the 26th International Conference on Software Engineering, pp. 86–95. IEEE Computer Society, Washington (2004)Google Scholar
  10. 10.
    Chang, J., Richardson, D.J., Sankar, S.: Structural specification-based testing with adl. In: ISSTA, pp. 62–70 (1996)Google Scholar
  11. 11.
  12. 12.
    DeMillo R.A., Lipton R.J. and Sayward F.G. (1978). Hints on test data selection: Help for the practicing programmer. IEEE Computer 11(4): 34–41 Google Scholar
  13. 13.
    Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: FME, pp. 268–284 (1993)Google Scholar
  14. 14.
    Do, H., Elbaum, S., Rothermel, G.: Infrastructure support for controlled experimentation with software testing and regression testing techniques. In: ISESE ’04: Proceedings of the 2004 International Symposium on Empirical Software Engineering, pp. 60–70. IEEE Computer Society, Washington (2004)Google Scholar
  15. 15.
    Fowler M. (1999). Refactoring: Improving the Design of Existing Code. Addison-Wesley, Reading Google Scholar
  16. 16.
    Frankl, P.G., Deng, Y.: Comparison of delivered reliability of branch, data flow and operational testing: a case study. In: ISSTA ’00: Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 124–134. ACM Press, New York (2000)Google Scholar
  17. 17.
    Frankl P.G. and Weiss S.N. (1993). An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Trans Softw. Eng. 19(8): 774–787 CrossRefGoogle Scholar
  18. 18.
    Frankl P.G. and Weyuker E.J. (1993). A formal analysis of the fault-detecting ability of testing methods. IEEE Trans Softw. Eng. 19(3): 202–213 CrossRefGoogle Scholar
  19. 19.
    Frankl P.G. and Weyuker E.J. (1993). Provable improvements on branch testing. IEEE Trans. Softw. Eng. 19(10): 962–975 CrossRefMathSciNetGoogle Scholar
  20. 20.
    Goodenough J.B. and Gerhart S.L. (1975). Toward a theory of test data selection. IEEE Trans. Software Eng. 1(2): 156–173 MathSciNetGoogle Scholar
  21. 21.
    Grieskamp, W., Gurevich, Y., Schulte, W., Veanes, M.: Generating finite state machines from abstract state machines. In: ISSTA ’02: Proceedings of the 2002 ACM SIGSOFT International Symposium on software Testing and Analysis, pp. 112–122. ACM Press, New York (2002)Google Scholar
  22. 22.
    Hamlet, R.: Theoretical comparison of testing methods. In: TAV3: Proceedings of the ACM SIGSOFT ’89 3rd Symposium on Software Testing, Analysis, and Verification, pp. 28–37. ACM Press, New York (1989)Google Scholar
  23. 23.
    Höst, M., Wohlin, C., Thelin, T.: Experimental context classification: incentives and experience of subjects. In: ICSE ’05: Proceedings of the 27th International Conference on Software Engineering, pp. 470–478 (2005)Google Scholar
  24. 24.
    Hutchins, M., Foster, H., Goradia, T., Ostrand T.: Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In ICSE ’94: Proceedings of the 16th International Conference on Software Engineering, pp. 191–200. IEEE Computer Society Press, Los Alamitos (1994)Google Scholar
  25. 25.
  26. 26.
    JUnit Home Page. http://www.junit.org
  27. 27.
    Juristo N. and Moreno A.M. (2001). Basics of Software Engineering Experimentation. Springer, Heidelberg MATHGoogle Scholar
  28. 28.
    Juristo N., Moreno A.M. and Vegas S. (2004). Reviewing 25 years of testing technique experiments. Empirical Softw. Engg. 9(1–2): 7–44 CrossRefGoogle Scholar
  29. 29.
    Juristo N., Moreno A.M. and Vegas S. (2004). Towards building a solid empirical body of knowledge in testing techniques. SIGSOFT Softw. Eng. Notes 29(5): 1–4 CrossRefGoogle Scholar
  30. 30.
    Juristo N., Moreno A.M., Vegas S. and Solari M. (2006). In search of what we experimentally know about unit testing. IEEE Softw. 23(6): 72–80 CrossRefGoogle Scholar
  31. 31.
    Kamsties, E., Lott, C.M.: An empirical evaluation of three defect-detection techniques. In: ESEC ’95: Proceedings of the 5th European Software Engineering Conference, pp. 362–383. Springer, London (1995)Google Scholar
  32. 32.
    Kapoor K. and Bowen J.P. (2004). Experimental evaluation of the tolerance for control-flow test criteria: Research articles. Softw. Test. Verif. Reliab. 14(3): 167–187 CrossRefGoogle Scholar
  33. 33.
    Kim, S., Clark, J., McDermid, J.: The rigorous generation of java mutation operators using hazop. In: Proceedings of ICSSEA, pp. 9–10 (1999)Google Scholar
  34. 34.
    Kitchenham B.A., Pfleeger S.L., Pickard L.M., Jones P.W., Hoaglin D.C., Emam K.E. and Rosenberg J. (2002). Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8): 721–734 CrossRefGoogle Scholar
  35. 35.
    Lingampally, R., Gupta, A., Jalote, P.: A multipurpose code coverage tool for java. In: HICSS ’07: Proceedings of 40th Annual Hawaii International Conference on System Sciences; minitrack on Automated Software Testing and Analysis: Techniques, Practices and Tools, pp. 261–270. IEEE Computer Society, Los Alamitos (2007)Google Scholar
  36. 36.
    Ma, Y.-S., Offutt, J., Kwon, Y.-R.: Mujava: a mutation system for java. In: ICSE ’06: Proceeding of the 28th International Conference on Software Engineering, pp. 827–830. ACM Press, New York (2006)Google Scholar
  37. 37.
    Miller, J.: Can results from software engineering experiments be safely combined? In: METRICS ’99: Proceedings of the 6th International Symposium on Software Metrics, pp. 152–158. IEEE Computer Society, Washington (1999)Google Scholar
  38. 38.
    Monopoly - RealEstate. http://open.ncsu.edu/se/monopoly/
  39. 39.
    Montgomery D.C. (2001). Design and Analysis of Experiments. Wiley, London Google Scholar
  40. 40.
  41. 41.
    Murray, L., Carrington, D.A., MacColl, I., McDonald, J., Strooper, P.A.: Formal derivation of finite state machines for class testing. In: ZUM, pp. 42–59 (1998)Google Scholar
  42. 42.
    MySQL Home Page. http://www.mysql.com/
  43. 43.
    Ntafos, S.C.: An evaluation of required element testing strategies. In: ICSE ’84: Proceedings of the 7th International Conference on Software Engineering, pp. 250–256. IEEE Press, Piscataway (1984)Google Scholar
  44. 44.
    Offutt A.J. (1992). Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1(1): 5–20 CrossRefGoogle Scholar
  45. 45.
    Offutt, A.J., Xiong, Y., Liu, S.: Criteria for generating specification-based tests. In: ICECCS ’99 : 5th International Conference on Engineering of Complex Computer Systems, pp. 119–129. IEEE Computer Society, Washington (1999)Google Scholar
  46. 46.
    Shull, F., Basili, V., Carver, J., Maldonado, J.C., Travassos, G.H., Mendonça, M., Fabbri, S.: Replicating software engineering experiments: Addressing the tacit knowledge problem. In: ISESE ’02: Proceedings of the 2002 International Symposium on Empirical Software Engineering, pp. 7–16. IEEE Computer Society, Washington (2002)Google Scholar
  47. 47.
    Sjoberg D.I.K., Hannay J.E., Hansen O., Kampenes V.B., Karahasanovic A., Liborg N.-K. and Rekdal A.C. (2005). A survey of controlled experiments in software engineering. IEEE Trans. Softw. Eng. 31(9): 733–753 CrossRefGoogle Scholar
  48. 48.
    Stocks P. and Carrington D.A. (1996). A framework for specification-based testing. IEEE Trans. Softw. Eng. 22(11): 777–793 CrossRefGoogle Scholar
  49. 49.
    Weyuker, E.: Can we measure software testing effectiveness? In: Metrics ’93: Proceedings of 1st International Symposium on Software Metrics, pp. 100–107. IEEE Computer Society, Washington (1993)Google Scholar
  50. 50.
    Weyuker, E.J., Weiss, S.N., Hamlet, D.: Comparison of program testing strategies. In: TAV4: Proceedings of the Symposium on Testing, Analysis, and Verification, pp. 1–10. ACM Press, New York (1991)Google Scholar
  51. 51.
    Wohlin C., Runeson P., Host M., Ohlsson M.C., Regnell B. and Wesslen A. (2000). Experimentation in Software Engineering—An Introduction. Kluwer, Dordrecht MATHGoogle Scholar
  52. 52.
    Wood, M., Roper, M., Brooks, A., Miller, J.: Comparing and combining software defect detection techniques: a replicated empirical study. In: ESEC ’97/FSE-5: Proceedings of the 6th European Conference/5th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 262–277. Springer, New York (1997)Google Scholar
  53. 53.
    Zhu H., Hall P.A.V. and May J.H.R. (1997). Software unit test coverage and adequacy. ACM Comput. Surv. 29(4): 366–427 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2007

Authors and Affiliations

  1. 1.Indian Institute of TechnologyKanpurIndia
  2. 2.Indian Institute of Technology DelhiDelhiIndia

Personalised recommendations