Advertisement

Empirical Software Engineering

, Volume 6, Issue 2, pp 111–142 | Cite as

Interface Mutation Test Adequacy Criterion: An Empirical Evaluation

  • MÁrcio Eduardo Delamaro
  • JosÉ Carlos Maldonado
  • Alberto Pasquini
  • Aditya P. Mathur
Article

Abstract

An experiment was conducted to evaluate an inter-procedural test adequacy criterion named Interface Mutation. Program SPACE, developed for the European Space Agency (ESA), was used in this experiment. The development record available for this program was used to find the faults uncovered during its development. Using this information the test process was reproduced starting with a version of SPACE containing several faults and then applying Interface Mutation. Thus we could evaluate the fault revealing effectiveness of Interface Mutation. Results from the experiment suggest that (a) the application of Interface Mutation favors the selection of fault revealing test cases when they exist and (b) Interface Mutation tends to select fault revealing test cases more efficiently than in the case where random selection is used.

Mutation testing interface mutation test adequacy criteria software testing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Acree, A. T. 1980. On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta, GA.Google Scholar
  2. Bache, R. 1997. The Effect of Fault Size on Testing. Journal of Software Testing, Verification and Reliability 7(3): 139-152.Google Scholar
  3. Beizer, B. 1990. Software Testing Techniques. 2nd Ed. Van Nostrand Reinhold Company, New York.Google Scholar
  4. Budd, T. A. 1980. Mutation Analysis of Program Test Data. PhD thesis, Yale University, New Haven, CT.Google Scholar
  5. Delamaro, M. E. 1997. Mutação de interface: Um Critério de Adequação Inter-procedimental para o Teste de Integração. Doctoral dissertation, Physics Institute of São Carlos, University of São Paulo, São Carlos, SP.Google Scholar
  6. Delamaro, M. E., and Maldonado, J. C. 1999. Interface Mutation: Assessing Testing Quality at Interprocedural Level. In: Proceedings of the International Conference of the Chilean Computer Society (SCCC99). Talca, Chile.Google Scholar
  7. Delamaro, M. E., Maldonado, J. C., and Mathur, A. P. 1996. Integration Testing Using Interface Mutation. In: Proceedings of the Seventh International Symposium of Software Reliability Engineering (ISSRE'96). White Plains, NY, pp. 112-121.Google Scholar
  8. Delamaro, M. E., Maldonado, J. C., and Mathur, A. P. 2001. Interface Mutation: An Approach for Integration Testing. IEEE Transactions on Software Engineering, 27(3), March 2001.Google Scholar
  9. Delamaro, M. E., J. C. Maldonado, A. Pasquini, and A. P. Mathur: 2000, 'Interface Mutation Test Adequacy Criterion: An Empirical Evaluation'. Tech report, www.din.uem.br/~delamaro/papers/spaceinternet.ps.Google Scholar
  10. DeMillo, R. A., Lipton, R. J., and Sayward, F. G. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer 11(4).Google Scholar
  11. DeMillo, R. A., and Offutt, A. J. 1991. Constraint Based Automatic Test Data Generation. IEEE Transactions on Software Engineering 17(9): 900-910.Google Scholar
  12. Frankl, P. G., and Weyuker, E. J. 1993. A Formal Analysis of the Fault-Detecting Ability of Testing Methods. IEEE Transactions on Software Engineering 19(3): 202-213.Google Scholar
  13. Haley, A., and Zweben, S. 1984. Development and Application of a White Box Approach to Integration Testing. The Journal of Systems and Software 4: 309-315.Google Scholar
  14. Harrold, M. J., and Soffa, M. L. 1991. Selecting and Using Data for Integration Test. IEEE Software 8(2): 58-65.Google Scholar
  15. IEEE Computer Society: 1990, 'Standard Glossary of Software Engineering Terminology', Standard 610.12, IEEE Press.Google Scholar
  16. IEEE Computer Society: 1994, 'Standard Classification for Software Errors, Faults and Failures'. Standard, IEEE Press.Google Scholar
  17. Jin, Z., and Offut, A. J. 1995. Integration Testing Based on Software Couplings. In: Proceedings of the Tenth Annual Conference on Computer Assurance (COMPASS 95). Gaithersburg, Maryland, pp. 13-23.Google Scholar
  18. Linnenkugel, U., and Müllerburg, M. 1990. Test Data Selection Criteria for (Software) Integration Testing. In: Proceedings of the First International Conference on Systems Integration. Mornstown, NJ, pp. 709-717.Google Scholar
  19. Mathur, A. P. 1991. Performance, Effectiveness and Reliability Issues in Software Testing. In: Proceeding of the Fifteenth Annual International Computer Software and Applications Conference. Tokio, Japan, pp. 604-605.Google Scholar
  20. Mathur, A. P., and Wong, W. E. 1994. An Empirical Comparison of Data Flow and Mutation Based Test Adequacy Criteria. Journal of Software Testing, Verification, and Reliability 4(1): 9-31.Google Scholar
  21. Morell, L. 1990. A Theory of Fault-Based Testing. IEEE Transactions on Software Engineering 16(8): 844-857.Google Scholar
  22. Offut, A. J., and Hayes, J. H. 1996. A Semantic Model of Program Faults. In: Proceedings of ISSTA'96. San Diego, CA.Google Scholar
  23. Offutt, A. J. 1989. Coupling Effect: Fact or Fiction. In: Proceedings of the Third Symposium on Software Testing, Analysis, and Verification. Key West, FL, pp. 131-140.Google Scholar
  24. Offutt, A. J. 1992. Investigations of The Software Testing Coupling Effect. ACM Transactions on Software Engineering Methodology 1(1): 3-18.Google Scholar
  25. Offutt, A. J., Lee, A., Rothermel, G., Untch, R. H., and Zapf, C. 1996. An Experimental Determination of Sufficient Mutant Operators. ACM Transactions on Software Engineering Methodology 5(2): 99-118.Google Scholar
  26. Offutt, A. J., Rothermel, G., and Zapf, C. 1993. An Experimental Evaluation of Selective Mutation. In: Proceedings of the Fifteenth International Conference on Software Engineering. Baltimore, MD, pp. 100-107.Google Scholar
  27. Rapps, S., and Weyuker, E. J., 1985. Selecting Software Test Data Using Data Flow Information. IEEE Transactions on Software Engineering SE-11(4): 367-375.Google Scholar
  28. Vincenzi, A. M. R. 1998. Subsídios para a Desenvolvimeto de Estratégias de Teste Baseadas na Técnica de Mutação. Master theses, SCE-ICMC-USP, São Carlos, SP.Google Scholar
  29. Voas, J. M., and Miller, K. W. 1992. The Revealing Power of a Test Case. Journal of Software Testing, Verification and Reliability 2(1): 25-42.Google Scholar
  30. Wah, K. S. H. T. 1995. Fault Coupling in Finite Bijective Functions. Journal of Software Testing Verification and Reliability 5(1): 3-47.Google Scholar
  31. Wong, W. E. 1993. On Mutation and Data Flow. PhD dissertation, Department of Computer Science, Purdue University, W. Lafayette, IN.Google Scholar
  32. Wong, W. E., Horgan, J. R., London, S., and Agrawal, H. 1997. Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application. In: Proceedings of the Eighth IEEE International Symposium on Software Reliability Engineering (ISSRE'97). Albuquerque, New Mexico, pp. 522-528.Google Scholar
  33. Wong, W. E., Horgan, J. R., London, S., and Mathur, A. P. 1995. Effect of Test Set Minimization on Fault Detection Effectiveness. In: Proceedings of the 17th IEEE International Conference on Software Engineering. Seattle, WA, pp. 41-50.Google Scholar
  34. Wong, W. E., Horgan, J. R., Mathur, A. P., and Pasquini, A. 1999. Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application. Journal of Systems and Software 48(2): 79-89.Google Scholar
  35. Wong, W. E., and Mathur A. P. 1995a. Fault Detection Effectiveness of Mutation and Data Flow Testing. Software Quality Journal 4(1): 69-83.Google Scholar
  36. Wong, W. E., and Mathur, A. P. 1995b. Reducing the Cost of Mutation Testing: An Empirical Study. The Journal of Systems and Software 31(3): 185-196.Google Scholar
  37. Wong, W. E., Mathur, A. P., and Maldonado, J. C. 1994. Mutation Versus All-uses: An Empirical Evaluation of Cost, Strength, and Effectiveness. In: Proceedings of the International Conference on Software Quality and Productivity. Hong Kong, pp. 258-265.Google Scholar
  38. Yang, M. C. K., Wong, W. E., and Pasquini, A. 1998. Applying Testability to Reliability Estimation. In: Proceedings of the Nineth IEEE International Symposium on Software Reliability Engineering (ISSRE'98). Paderborn, Germany, pp. 90-99.Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • MÁrcio Eduardo Delamaro
    • 1
  • JosÉ Carlos Maldonado
    • 2
  • Alberto Pasquini
    • 3
  • Aditya P. Mathur
    • 4
  1. 1.Departamento de InformáticaUniversidade Estadual de MaringáMaringáPR, Brazil
  2. 2.Universidade de São PauloSão Carlos, SPBrazil
  3. 3.Robotics and Information Technology DivisionItalian Research Agency for New Technology, Energy and Environment (ENEA)RomaItaly
  4. 4.Department of ComputerSciences Purdue UniversityUSA

Personalised recommendations