Skip to main content
Log in

Interface Mutation Test Adequacy Criterion: An Empirical Evaluation

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

An experiment was conducted to evaluate an inter-procedural test adequacy criterion named Interface Mutation. Program SPACE, developed for the European Space Agency (ESA), was used in this experiment. The development record available for this program was used to find the faults uncovered during its development. Using this information the test process was reproduced starting with a version of SPACE containing several faults and then applying Interface Mutation. Thus we could evaluate the fault revealing effectiveness of Interface Mutation. Results from the experiment suggest that (a) the application of Interface Mutation favors the selection of fault revealing test cases when they exist and (b) Interface Mutation tends to select fault revealing test cases more efficiently than in the case where random selection is used.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Acree, A. T. 1980. On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta, GA.

    Google Scholar 

  • Bache, R. 1997. The Effect of Fault Size on Testing. Journal of Software Testing, Verification and Reliability 7(3): 139-152.

    Google Scholar 

  • Beizer, B. 1990. Software Testing Techniques. 2nd Ed. Van Nostrand Reinhold Company, New York.

    Google Scholar 

  • Budd, T. A. 1980. Mutation Analysis of Program Test Data. PhD thesis, Yale University, New Haven, CT.

    Google Scholar 

  • Delamaro, M. E. 1997. Mutação de interface: Um Critério de Adequação Inter-procedimental para o Teste de Integração. Doctoral dissertation, Physics Institute of São Carlos, University of São Paulo, São Carlos, SP.

    Google Scholar 

  • Delamaro, M. E., and Maldonado, J. C. 1999. Interface Mutation: Assessing Testing Quality at Interprocedural Level. In: Proceedings of the International Conference of the Chilean Computer Society (SCCC99). Talca, Chile.

  • Delamaro, M. E., Maldonado, J. C., and Mathur, A. P. 1996. Integration Testing Using Interface Mutation. In: Proceedings of the Seventh International Symposium of Software Reliability Engineering (ISSRE'96). White Plains, NY, pp. 112-121.

  • Delamaro, M. E., Maldonado, J. C., and Mathur, A. P. 2001. Interface Mutation: An Approach for Integration Testing. IEEE Transactions on Software Engineering, 27(3), March 2001.

  • Delamaro, M. E., J. C. Maldonado, A. Pasquini, and A. P. Mathur: 2000, 'Interface Mutation Test Adequacy Criterion: An Empirical Evaluation'. Tech report, www.din.uem.br/~delamaro/papers/spaceinternet.ps.

  • DeMillo, R. A., Lipton, R. J., and Sayward, F. G. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer 11(4).

  • DeMillo, R. A., and Offutt, A. J. 1991. Constraint Based Automatic Test Data Generation. IEEE Transactions on Software Engineering 17(9): 900-910.

    Google Scholar 

  • Frankl, P. G., and Weyuker, E. J. 1993. A Formal Analysis of the Fault-Detecting Ability of Testing Methods. IEEE Transactions on Software Engineering 19(3): 202-213.

    Google Scholar 

  • Haley, A., and Zweben, S. 1984. Development and Application of a White Box Approach to Integration Testing. The Journal of Systems and Software 4: 309-315.

    Google Scholar 

  • Harrold, M. J., and Soffa, M. L. 1991. Selecting and Using Data for Integration Test. IEEE Software 8(2): 58-65.

    Google Scholar 

  • IEEE Computer Society: 1990, 'Standard Glossary of Software Engineering Terminology', Standard 610.12, IEEE Press.

  • IEEE Computer Society: 1994, 'Standard Classification for Software Errors, Faults and Failures'. Standard, IEEE Press.

  • Jin, Z., and Offut, A. J. 1995. Integration Testing Based on Software Couplings. In: Proceedings of the Tenth Annual Conference on Computer Assurance (COMPASS 95). Gaithersburg, Maryland, pp. 13-23.

  • Linnenkugel, U., and Müllerburg, M. 1990. Test Data Selection Criteria for (Software) Integration Testing. In: Proceedings of the First International Conference on Systems Integration. Mornstown, NJ, pp. 709-717.

  • Mathur, A. P. 1991. Performance, Effectiveness and Reliability Issues in Software Testing. In: Proceeding of the Fifteenth Annual International Computer Software and Applications Conference. Tokio, Japan, pp. 604-605.

  • Mathur, A. P., and Wong, W. E. 1994. An Empirical Comparison of Data Flow and Mutation Based Test Adequacy Criteria. Journal of Software Testing, Verification, and Reliability 4(1): 9-31.

    Google Scholar 

  • Morell, L. 1990. A Theory of Fault-Based Testing. IEEE Transactions on Software Engineering 16(8): 844-857.

    Google Scholar 

  • Offut, A. J., and Hayes, J. H. 1996. A Semantic Model of Program Faults. In: Proceedings of ISSTA'96. San Diego, CA.

  • Offutt, A. J. 1989. Coupling Effect: Fact or Fiction. In: Proceedings of the Third Symposium on Software Testing, Analysis, and Verification. Key West, FL, pp. 131-140.

  • Offutt, A. J. 1992. Investigations of The Software Testing Coupling Effect. ACM Transactions on Software Engineering Methodology 1(1): 3-18.

    Google Scholar 

  • Offutt, A. J., Lee, A., Rothermel, G., Untch, R. H., and Zapf, C. 1996. An Experimental Determination of Sufficient Mutant Operators. ACM Transactions on Software Engineering Methodology 5(2): 99-118.

    Google Scholar 

  • Offutt, A. J., Rothermel, G., and Zapf, C. 1993. An Experimental Evaluation of Selective Mutation. In: Proceedings of the Fifteenth International Conference on Software Engineering. Baltimore, MD, pp. 100-107.

  • Rapps, S., and Weyuker, E. J., 1985. Selecting Software Test Data Using Data Flow Information. IEEE Transactions on Software Engineering SE-11(4): 367-375.

    Google Scholar 

  • Vincenzi, A. M. R. 1998. Subsídios para a Desenvolvimeto de Estratégias de Teste Baseadas na Técnica de Mutação. Master theses, SCE-ICMC-USP, São Carlos, SP.

    Google Scholar 

  • Voas, J. M., and Miller, K. W. 1992. The Revealing Power of a Test Case. Journal of Software Testing, Verification and Reliability 2(1): 25-42.

    Google Scholar 

  • Wah, K. S. H. T. 1995. Fault Coupling in Finite Bijective Functions. Journal of Software Testing Verification and Reliability 5(1): 3-47.

    Google Scholar 

  • Wong, W. E. 1993. On Mutation and Data Flow. PhD dissertation, Department of Computer Science, Purdue University, W. Lafayette, IN.

    Google Scholar 

  • Wong, W. E., Horgan, J. R., London, S., and Agrawal, H. 1997. Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application. In: Proceedings of the Eighth IEEE International Symposium on Software Reliability Engineering (ISSRE'97). Albuquerque, New Mexico, pp. 522-528.

  • Wong, W. E., Horgan, J. R., London, S., and Mathur, A. P. 1995. Effect of Test Set Minimization on Fault Detection Effectiveness. In: Proceedings of the 17th IEEE International Conference on Software Engineering. Seattle, WA, pp. 41-50.

  • Wong, W. E., Horgan, J. R., Mathur, A. P., and Pasquini, A. 1999. Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application. Journal of Systems and Software 48(2): 79-89.

    Google Scholar 

  • Wong, W. E., and Mathur A. P. 1995a. Fault Detection Effectiveness of Mutation and Data Flow Testing. Software Quality Journal 4(1): 69-83.

    Google Scholar 

  • Wong, W. E., and Mathur, A. P. 1995b. Reducing the Cost of Mutation Testing: An Empirical Study. The Journal of Systems and Software 31(3): 185-196.

    Google Scholar 

  • Wong, W. E., Mathur, A. P., and Maldonado, J. C. 1994. Mutation Versus All-uses: An Empirical Evaluation of Cost, Strength, and Effectiveness. In: Proceedings of the International Conference on Software Quality and Productivity. Hong Kong, pp. 258-265.

  • Yang, M. C. K., Wong, W. E., and Pasquini, A. 1998. Applying Testability to Reliability Estimation. In: Proceedings of the Nineth IEEE International Symposium on Software Reliability Engineering (ISSRE'98). Paderborn, Germany, pp. 90-99.

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Delamaro, M.E., Maldonado, J.C., Pasquini, A. et al. Interface Mutation Test Adequacy Criterion: An Empirical Evaluation. Empirical Software Engineering 6, 111–142 (2001). https://doi.org/10.1023/A:1011429104252

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1011429104252

Navigation