Skip to main content
SpringerLink
Log in

A Mutation-Based Approach for Testing AsmetaL Specifications

  • Research Article - Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

The detection of specification errors can help reduce the cost and risk of software development because uncorrected defects in specifications will propagate to code, thus adversely affecting the quality of the end product. Mutation testing is a well-established fault-based technique for assessing and improving the quality of test suites. Mutation testing can be applied at different levels of abstraction, e.g., the unit level, the integration level, and the specification level. In this paper, we propose a suite of AsmetaL-specific mutation operators, classified into four categories. The proposed operators are used to assess the adequacy of test suites generated using the ATGT tool, according to various test coverage criteria. We demonstrate the applicability of our approach through eight publicly available AsmetaL case studies. The results of the case studies show that our proposed mutation operators can be used to compare different AsmetaL-based test coverage criteria and successfully detect inadequacies in test suites.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Morell L.J.: A theory of fault-based testing. IEEE Trans. Softw. Eng. 16(8), 844–857 (1990) Doi: 10.1109/32.57623

    Article  Google Scholar 

  2. DeMillo R.A., Lipton R.J., Sayward F.G.: Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41 (1978)

    Article  Google Scholar 

  3. Ammann, P.; Black, P.E.: A specification-based coverage metric to evaluate test sets. In: Proceddings of the 4th IEEE International Symposium on High-Assurance Systems Engineering (HASE), 7– 19 November 1999, Washington, D.C, USA, pp. 239–248. IEEE Computer Society (1999)

  4. Jia Y., Harman M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)

    Article  Google Scholar 

  5. Fabbri, S.P.F.; Delamaro,M.; Maldonado, J.;Masiero, P.: Mutation analysis testing for finite state machines. In: Proceedings of the 5th International Symposium on Software Reliability Engineering, Monterey, California, USA, pp. 220–229. IEEE Computer Society (1994)

  6. Fabbri, S.C.P.F.; Maldonado, J.C.; Sugeta, T.; Masiero, P.C.: Mutation testing applied to validate specifications based on statecharts. In: Proceedings of the 10th International Symposium on Software Reliability Engineering (ISSRE), Boca Raton, Florida, USA, pp. 210–219. IEEE Computer Society (1999)

  7. Batth, S.; Vieira, E.; Cavalli, A.; Uyar, M.U.: Specification of timed EFSM fault models in SDL. In: Derrick, J., Vain J. (eds.) Formal Techniques for Networked and Distributed Systems (FORTE), LNCS, vol. 4574, pp. 50–65. Springer, Berlin (2007). doi:10.1007/978-3-540-73196-2_4

  8. Börger E., Stark R.F.: Abstract State Machines: A Method for High-Level System Design and Analysis. Springer, Secaucus (2003)

    Book  Google Scholar 

  9. Hassine, J.: Design and classification of mutation operators for abstract state machines. Int. J. Adv. Softw. 6(1), 80–91 (2013). http://www.thinkmind.org/index.php?view=article&articleid=soft_v6_n12_2013_7

  10. Scandurra, P.; Gargantini, A.; Genovese, C.; Genovese, T.; Riccobene, E.: A concrete syntax derived from the abstract state machine metamodel. In: Proceedings of the 12th International Workshop on Abstract State Machines (ASM 2005), pp. 345–368. Springer, Paris (2005)

  11. Alkrarha, O.; Hassine, J.: MuAsmetaL: An experimental mutation system for AsmetaL. In: Proceedings of the 12th International Conference on Information Technology: New Generations (ITNG), pp. 421–426. IEEE Computer Society (2015). doi:10.1109/ITNG.2015.74

  12. Gargantini, A.; Calvagna, A.; Riccobene, E.; Rinzivillo, S.; Galati, S.: ATGT: ASM Tests Generation Tool. http://cs.unibg.it/gargantini/software/atgt/ (2013)

  13. Carioni, A.; Gargantini, A.; Riccobene, E.; Scandurra, P.: A scenario-based validation language for ASMs. In: Börger, E., Butler, M., Bowen, J., Boca, P. (eds.) Abstract State Machines, B and Z, LNCS, vol. 5238, pp. 71–84. Springer, Berlin (2008). doi:10.1007/978-3-540-87603-8_7

  14. Formal Methods and SE Laboratory—University of Milan: AsmetaL compiler. http://asmeta.sourceforge.net/download/asmetalc.html (2014). Last accessed May 2015

  15. Gargantini, A.; Riccobene, E.; Scandurra, P.: Examples of AsmetaL specification. http://cs.unibg.it/gargantini/software/atgt/examples.zip (2014)

  16. Gurevich, Y.: Specification and validation methods. Evolving Algebras 1993: Lipari Guide, pp. 9–36. Oxford University Press, Inc., New York, NY (1995)

  17. AsmetaL: AsmM Concrete Syntax (AsmetaL) v.2.0.0. http://fmse.di.unimi.it/asmeta/download/AsmetaL_quickguide.html (2014)

  18. Formal Methods and SE Laboratory—University of Milan: AsmetaL EBNF. http://fmse.di.unimi.it/asmeta/download/AsmetaL_EBNF.html. Last accessed 03 May 2015

  19. Gargantini A., Riccobene E.: ASM-Based testing: coverage criteria and automatic test sequence. J. Univers. Comput. Sci. 7(11), 1050–1067 (2001)

    Google Scholar 

  20. Chilenski J.J., Miller S.P.: Applicability of modified condition/decision coverage to software testing. Softw. Eng. J. 9(5), 193 (1994)

    Article  Google Scholar 

  21. Gargantini, A.: Using model checking to generate fault detecting tests. In: Gurevich, Y., Meyer, B. (eds.) Tests and Proofs, Lecture Notes in Computer Science, vol. 4454, pp. 189–206. Springer, Berlin (2007). doi:10.1007/978-3-540-73770-4_11

  22. Calvagna, A.; Gargantini, A.: A logic-based approach to combinatorial testing with constraints. In: Beckert, B., Hähnle, R. (eds.) Tests and Proofs, Lecture Notes in Computer Science, vol. 4966, pp. 66–83. Springer, Berlin (2008). doi:10.1007/978-3-540-79124-9_6

  23. Calvagna, A.; Gargantini, A.: Combining satisfiability solving and heuristics to constrained combinatorial interaction testing. In: Dubois, C. (ed.) Tests and Proofs, Lecture Notes in Computer Science, vol. 5668, pp. 27–42. Springer, Berlin (2009). doi:10.1007/978-3-642-02949-3_4

  24. Ammann P., Offutt J.: Introduction to Software Testing, 1 edn. Cambridge University Press, New York, NY (2008)

    Book  MATH  Google Scholar 

  25. Jedlitschka, A.; Ciolkowski, M.; Pfahl, D.: Reporting experiments in software engineering. In: Shull, F., Singer, J., Sjøberg, D. (eds.) Guide to Advanced Empirical Software Engineering, pp. 201–228. Springer, London (2008). doi:10.1007/978-1-84800-044-5_8

  26. Grieskamp,W.; Gurevich, Y.; Schulte,W.; Veanes,M.: Generating finite statemachines fromAbstract StateMachines. In: Proceedings of the International Symposium on Software Testing and Analysis (ISSTA) July 22–24, Roma, Italy, pp. 112–122. ACM (2002)

  27. Barnett, M.; Grieskamp, W.; Nachmanson, L.; Schulte, W.; Tillmann, N.;Veanes,M.: Towards a tool environment for model-based testing with AsmL. In: Formal Approaches to Software Testing (FATES). LNCS, vol. 2931, pp. 252–266. Springer, (2003)

  28. AsmL: Microsoft Research: The Abstract State Machine Language. http://research.microsoft.com/en-us/projects/asml/ (2006)

  29. Veanes M., Bjørner N., Gurevich Y., Schulte W.: Symbolic bounded model checking of abstract state machines. Int. J. Softw. Inform. 3(2-3), 149–170 (2009)

    Google Scholar 

  30. Winter K.: Model checking for abstract state machines. J. Univers. Comput. Sci. 3(5), 689–701 (1997)

    MATH  Google Scholar 

  31. Arcaini, P.; Gargantini, A.; Riccobene, E.: AsmetaSMV: a way to link high-level ASM models to low-level NuSMV specifications. In: Abstract State Machines, Alloy, B and Z (ABZ). Second International Conference, ABZ 2010, Orford, QC, Canada, February 22–25, 2010, Proceedings. LNCS, vol. 5977, pp. 61–74. Springer, Berlin, Heidelberg (2010)

  32. Gargantini, A.; Riccobene, E.; Rinzivillo, S.: Using Spin to generate tests from ASM specifications. In: Börger, E., Gargantini, A., Riccobene, E. (eds.) Abstract State Machines, LNCS, vol. 2589, pp. 263–277. Springer, Berlin (2003). doi:10.1007/3-540-36498-6_15

  33. Hassine, J.: Abstract state machines mutation operators. In: The Seventh International Conference on Software Engineering Advances (ICSEA 2012), Lisbon, pp. 436–441. IARIA (2012). 18–23 Nov. http://www.thinkmind.org/index.php?view=article&articleid=icsea_2012_15_30_10395

  34. Farahbod R., Gervasi V., Glässer U.: CoreASM: an extensible ASM execution engine. Fundamenta Informaticae 77, 71–103 (2007)

    MATH  MathSciNet  Google Scholar 

  35. Wright, H.K.; Kim, M.; Perry, D.E.: Validity concerns in software engineering research. In: Roman, G.C., Sullivan, K.J. (eds.) FoSER Santa Fe, New Mexico, USA, pp. 411–414. ACM (2010)

  36. Madeyski L.: Impact of pair programming on thoroughness and fault detection effectiveness of unit test suites. Softw Process Improv. Pract. 13(3), 281–295 (2008) Doi:10.1002/spip.382

    Article  Google Scholar 

  37. Offutt A.J., Lee A., Rothermel G., Untch R.H., Zapf C.: An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. 5(2), 99–118 (1996) Doi:10.1145/227607.227610

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information and Computer Science, King Fahd University of Petroleum and Minerals, Dhahran, KSA

    Jameleddine Hassine & Osama Alkrarha

Authors
  1. Jameleddine Hassine
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Osama Alkrarha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jameleddine Hassine.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hassine, J., Alkrarha, O. A Mutation-Based Approach for Testing AsmetaL Specifications. Arab J Sci Eng 40, 3523–3544 (2015). https://doi.org/10.1007/s13369-015-1832-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-015-1832-5

Keywords

Search

Navigation