Abstract
An experiment was conducted to evaluate an inter-procedural test adequacy criterion named Interface Mutation. Program SPACE, developed for the European Space Agency (ESA), was used in this experiment. The development record available for this program was used to find the faults uncovered during its development. Using this information the test process was reproduced starting with a version of SPACE containing several faults and then applying Interface Mutation. Thus we could evaluate the fault revealing effectiveness of Interface Mutation. Results from the experiment suggest that (a) the application of Interface Mutation favors the selection of fault revealing test cases when they exist and (b) Interface Mutation tends to select fault revealing test cases more efficiently than in the case where random selection is used.
Similar content being viewed by others
References
Acree, A. T. 1980. On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta, GA.
Bache, R. 1997. The Effect of Fault Size on Testing. Journal of Software Testing, Verification and Reliability 7(3): 139-152.
Beizer, B. 1990. Software Testing Techniques. 2nd Ed. Van Nostrand Reinhold Company, New York.
Budd, T. A. 1980. Mutation Analysis of Program Test Data. PhD thesis, Yale University, New Haven, CT.
Delamaro, M. E. 1997. Mutação de interface: Um Critério de Adequação Inter-procedimental para o Teste de Integração. Doctoral dissertation, Physics Institute of São Carlos, University of São Paulo, São Carlos, SP.
Delamaro, M. E., and Maldonado, J. C. 1999. Interface Mutation: Assessing Testing Quality at Interprocedural Level. In: Proceedings of the International Conference of the Chilean Computer Society (SCCC99). Talca, Chile.
Delamaro, M. E., Maldonado, J. C., and Mathur, A. P. 1996. Integration Testing Using Interface Mutation. In: Proceedings of the Seventh International Symposium of Software Reliability Engineering (ISSRE'96). White Plains, NY, pp. 112-121.
Delamaro, M. E., Maldonado, J. C., and Mathur, A. P. 2001. Interface Mutation: An Approach for Integration Testing. IEEE Transactions on Software Engineering, 27(3), March 2001.
Delamaro, M. E., J. C. Maldonado, A. Pasquini, and A. P. Mathur: 2000, 'Interface Mutation Test Adequacy Criterion: An Empirical Evaluation'. Tech report, www.din.uem.br/~delamaro/papers/spaceinternet.ps.
DeMillo, R. A., Lipton, R. J., and Sayward, F. G. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer 11(4).
DeMillo, R. A., and Offutt, A. J. 1991. Constraint Based Automatic Test Data Generation. IEEE Transactions on Software Engineering 17(9): 900-910.
Frankl, P. G., and Weyuker, E. J. 1993. A Formal Analysis of the Fault-Detecting Ability of Testing Methods. IEEE Transactions on Software Engineering 19(3): 202-213.
Haley, A., and Zweben, S. 1984. Development and Application of a White Box Approach to Integration Testing. The Journal of Systems and Software 4: 309-315.
Harrold, M. J., and Soffa, M. L. 1991. Selecting and Using Data for Integration Test. IEEE Software 8(2): 58-65.
IEEE Computer Society: 1990, 'Standard Glossary of Software Engineering Terminology', Standard 610.12, IEEE Press.
IEEE Computer Society: 1994, 'Standard Classification for Software Errors, Faults and Failures'. Standard, IEEE Press.
Jin, Z., and Offut, A. J. 1995. Integration Testing Based on Software Couplings. In: Proceedings of the Tenth Annual Conference on Computer Assurance (COMPASS 95). Gaithersburg, Maryland, pp. 13-23.
Linnenkugel, U., and Müllerburg, M. 1990. Test Data Selection Criteria for (Software) Integration Testing. In: Proceedings of the First International Conference on Systems Integration. Mornstown, NJ, pp. 709-717.
Mathur, A. P. 1991. Performance, Effectiveness and Reliability Issues in Software Testing. In: Proceeding of the Fifteenth Annual International Computer Software and Applications Conference. Tokio, Japan, pp. 604-605.
Mathur, A. P., and Wong, W. E. 1994. An Empirical Comparison of Data Flow and Mutation Based Test Adequacy Criteria. Journal of Software Testing, Verification, and Reliability 4(1): 9-31.
Morell, L. 1990. A Theory of Fault-Based Testing. IEEE Transactions on Software Engineering 16(8): 844-857.
Offut, A. J., and Hayes, J. H. 1996. A Semantic Model of Program Faults. In: Proceedings of ISSTA'96. San Diego, CA.
Offutt, A. J. 1989. Coupling Effect: Fact or Fiction. In: Proceedings of the Third Symposium on Software Testing, Analysis, and Verification. Key West, FL, pp. 131-140.
Offutt, A. J. 1992. Investigations of The Software Testing Coupling Effect. ACM Transactions on Software Engineering Methodology 1(1): 3-18.
Offutt, A. J., Lee, A., Rothermel, G., Untch, R. H., and Zapf, C. 1996. An Experimental Determination of Sufficient Mutant Operators. ACM Transactions on Software Engineering Methodology 5(2): 99-118.
Offutt, A. J., Rothermel, G., and Zapf, C. 1993. An Experimental Evaluation of Selective Mutation. In: Proceedings of the Fifteenth International Conference on Software Engineering. Baltimore, MD, pp. 100-107.
Rapps, S., and Weyuker, E. J., 1985. Selecting Software Test Data Using Data Flow Information. IEEE Transactions on Software Engineering SE-11(4): 367-375.
Vincenzi, A. M. R. 1998. Subsídios para a Desenvolvimeto de Estratégias de Teste Baseadas na Técnica de Mutação. Master theses, SCE-ICMC-USP, São Carlos, SP.
Voas, J. M., and Miller, K. W. 1992. The Revealing Power of a Test Case. Journal of Software Testing, Verification and Reliability 2(1): 25-42.
Wah, K. S. H. T. 1995. Fault Coupling in Finite Bijective Functions. Journal of Software Testing Verification and Reliability 5(1): 3-47.
Wong, W. E. 1993. On Mutation and Data Flow. PhD dissertation, Department of Computer Science, Purdue University, W. Lafayette, IN.
Wong, W. E., Horgan, J. R., London, S., and Agrawal, H. 1997. Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application. In: Proceedings of the Eighth IEEE International Symposium on Software Reliability Engineering (ISSRE'97). Albuquerque, New Mexico, pp. 522-528.
Wong, W. E., Horgan, J. R., London, S., and Mathur, A. P. 1995. Effect of Test Set Minimization on Fault Detection Effectiveness. In: Proceedings of the 17th IEEE International Conference on Software Engineering. Seattle, WA, pp. 41-50.
Wong, W. E., Horgan, J. R., Mathur, A. P., and Pasquini, A. 1999. Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application. Journal of Systems and Software 48(2): 79-89.
Wong, W. E., and Mathur A. P. 1995a. Fault Detection Effectiveness of Mutation and Data Flow Testing. Software Quality Journal 4(1): 69-83.
Wong, W. E., and Mathur, A. P. 1995b. Reducing the Cost of Mutation Testing: An Empirical Study. The Journal of Systems and Software 31(3): 185-196.
Wong, W. E., Mathur, A. P., and Maldonado, J. C. 1994. Mutation Versus All-uses: An Empirical Evaluation of Cost, Strength, and Effectiveness. In: Proceedings of the International Conference on Software Quality and Productivity. Hong Kong, pp. 258-265.
Yang, M. C. K., Wong, W. E., and Pasquini, A. 1998. Applying Testability to Reliability Estimation. In: Proceedings of the Nineth IEEE International Symposium on Software Reliability Engineering (ISSRE'98). Paderborn, Germany, pp. 90-99.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Delamaro, M.E., Maldonado, J.C., Pasquini, A. et al. Interface Mutation Test Adequacy Criterion: An Empirical Evaluation. Empirical Software Engineering 6, 111–142 (2001). https://doi.org/10.1023/A:1011429104252
Issue Date:
DOI: https://doi.org/10.1023/A:1011429104252