Using Model Checking to Generate Fault Detecting Tests

  • Angelo Gargantini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4454)


We present a technique which generates from Abstract State Machines specifications a set of test sequences capable to uncover specific fault classes. The notion of test goal is introduced as a state predicate denoting the detection condition for a particular fault. Tests are generated by forcing a model checker to produce counter examples which cover the test goals. We introduce a technique for the evaluation of the fault detection capability of a test set. We report some experimental results which validate the method, compare the fault adequacy criteria with some classical structural coverage criteria and show an empirical cross coverage among faults.


model based testing fault based testing Abstract State Machines test 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Akers, S.B.: On a theory of boolean functions. Journal Society Industrial Applied Mathematics 7(4), 487–498 (1959)MATHCrossRefGoogle Scholar
  2. 2.
    Ammann, P., Black, P.E., Ding, W.: Model checkers in software testing. Technical Report NIST-IR 6777, National Institute of Standards and Technology (2002)Google Scholar
  3. 3.
    Ammann, P., Black, P.E., Majurski, W.: Using model checking to generate tests from specifications. In: ICFEM 1998. 2nd IEEE International Conference on Formal Engineering Methods, Brisbane, Australia, December 1998, p. 46. IEEE Computer Society Press, Los Alamitos (1998)Google Scholar
  4. 4.
    Ammann, P.E., Black, P.E.: A specification-based coverage metric to evaluate test sets. International Journal of Reliability, Quality and Safety Engineering 8(4), 275–300 (2001)CrossRefGoogle Scholar
  5. 5.
    Atlee, J.M., Buckley, M.A.: A logic-model semantics for SCR software requirements. In: ISSTA 1996. Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis, pp. 280–292. ACM Press, New York, USA (1996)CrossRefGoogle Scholar
  6. 6.
    Barnett, M., Schulte, W.: The ABCs of specification: AsmL, behavior, and components. Informatica 25(4), 517–526 (2001)MATHGoogle Scholar
  7. 7.
    Beyer, D., Chlipala, A.J., Henzinger, T., Jhala, R., Majumdar, R.: Generating tests from counterexamples. In: ICSE 2004. Proc. International Conference on Software Engineering, Edinburgh, May 2004, pp. 326–335. IEEE CS Press, Los Alamitos (2004)Google Scholar
  8. 8.
    Black, P.E., Okun, V., Yesha, Y.: Mutation of model checker specifications for test generation and evaluation. In: Wong, W.E. (ed.) Mutation Testing for the New Century, proc. of Mutation 2000, October 2000, pp. 14–20. Kluwer Academic Publishers, Dordrecht (2000)Google Scholar
  9. 9.
    Börger, E., Stärk, R.: Abstract State Machines: A Method for High-Level System Design and Analysis. Springer, Heidelberg (2003)MATHGoogle Scholar
  10. 10.
    Chang, J., Richardson, D.J.: Structural specification-based testing: Automated support and experimental evaluation. In: Nierstrasz, O., Lemoine, M. (eds.) Software Engineering - ESEC/FSE 1999. LNCS, vol. 1687, pp. 285–302. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  11. 11.
    Chen, T.Y., Lau, M.F.: Test case selection strategies based on boolean specifications. Softw. Test., Verif. Reliab. 11(3), 165–180 (2001)CrossRefGoogle Scholar
  12. 12.
    Chilenski, J., Richey, L.A.: Definition for a masking form of modified condition decision coverage (mcdc). Technical report, Boeing, Seattle WA (1997)Google Scholar
  13. 13.
    Courtois, P.-J., Parnas, D.L.: Documentation for safety critical software. In: ICSE 1993. Proc. 15th Int’l Conf. on Softw. Eng., Baltimore, MD, pp. 315–323 (1993)Google Scholar
  14. 14.
    DeMillo, R.A., Guindi, D.S., King, K.N., McCracken, W.M., Offutt, A.J.: An extended overview of the Mothra software testing environment. In: Proceedings of the Second Workshop on Testing, Analysis, and Verification, pp. 142–151. IEEE Computer Society Press, Los Alamitos (1988)CrossRefGoogle Scholar
  15. 15.
    Dupuy, A., Leveson, N.: An empirical evaluation of the mc/dc coverage criterion on the hete-2 satellite software. In: The 19th Digital Avionics Systems Conferences. Proceedings DASC (2000)Google Scholar
  16. 16.
    Engels, A., Feijs, L., Mauw, S.: Test generation for intelligent networks using model checking. In: Brinksma, E. (ed.) TACAS 1997. LNCS, vol. 1217, pp. 384–398. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  17. 17.
    Frankl, P.G., Weyuker, E.J.: A formal analysis of the fault-detecting ability of testing methods. IEEE Transactions on Software Engineering 19(3), 202–213 (1993)CrossRefGoogle Scholar
  18. 18.
    Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. In: Nierstrasz, O., Lemoine, M. (eds.) Software Engineering - ESEC/FSE 1999. LNCS, vol. 1687, pp. 6–10. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  19. 19.
    Gargantini, A., Riccobene, E.: ASM-based testing: Coverage criteria and automatic test sequence generation. Journal of Universal Computer Science 7(11), 1050–1067 (2001)Google Scholar
  20. 20.
    Gargantini, A., Riccobene, E., Rinzivillo, S.: Using Spin to generate tests from ASM specifications. In: Börger, E., Gargantini, A., Riccobene, E. (eds.) ASM 2003. LNCS, vol. 2589, pp. 263–277. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  21. 21.
    Hamon, G., de Moura, L.M., Rushby, J.M.: Generating efficient test sets with a model checker. In: SEFM 2004. 2nd International Conference on Software Engineering and Formal Methods, Beijing, China, September 28-30, 2004, pp. 261–270 (2004)Google Scholar
  22. 22.
    Heimdahl, M.P., George, D.: Test-suite reduction for model based tests: Effects on test quality and implications for testing. In: Automated Software Engineering, Linz, Austria (September 2004)Google Scholar
  23. 23.
    Hierons, R.M.: Comparing test sets and criteria in the presence of test hypotheses and fault domains. ACM Trans. Softw. Eng. Methodol. 11(4), 427–448 (2002)CrossRefGoogle Scholar
  24. 24.
    Holzmann, G.J.: The model checker SPIN. IEEE Transactions on Software Engineering 23(5), 279–295 (1997)CrossRefMathSciNetGoogle Scholar
  25. 25.
    Hong, H.S., Cha, S.D., Lee, I., Sokolsky, O., Ural, H.: Data flow testing as model checking. In: ICSE 2003, Portland, Oregon, (May 3-10, 2003)Google Scholar
  26. 26.
    Hong, H.S., Lee, I., Sokolsky, O., Cha, S.D.: Automatic test generation from statecharts using model checking. In: Proceedings of FATES 2001, Workshop on Formal Approaches to Testing of Software, August 2001. BRICS Notes Series, vol. NS-01-4, pp. 15–30 (2001)Google Scholar
  27. 27.
    IEEE: IEEE Standard Glossary of Software Engineering Terminology. Institute of Electrical and Electronics Engineers, 610.12Google Scholar
  28. 28.
    Kapoor, K., Bowen, J.P.: Ordering mutants to minimise test effort in mutation testing. In: Grabowski, J., Nielsen, B. (eds.) FATES 2004. LNCS, vol. 3395, pp. 195–209. Springer, Heidelberg (2005)Google Scholar
  29. 29.
    Kapoor, K., Bowen, J.P.: A formal analysis of MCDC and RCDC test criteria. Softw. Test. Verif. Reliab. 15(1), 21–40 (2005)CrossRefGoogle Scholar
  30. 30.
    Kim, Y.G., Hong, H.S., Cho, S.M., Bae, D.H., Cha, S.D.: Test cases generation from UML state diagrams. IEE Proceedings - Software 146(4), 187–192 (1999)CrossRefGoogle Scholar
  31. 31.
    Kuhn, D.R.: Fault classes and error detection capability of specification-based testing. ACM Transactions on Software Engineering and Methodology 8(4), 411–424 (1999)CrossRefGoogle Scholar
  32. 32.
    Lau, M.F., Yu, Y.-T.: An extended fault class hierarchy for specification-based testing. ACM Trans. Softw. Eng. Methodol. 14(3), 247–276 (2005)CrossRefGoogle Scholar
  33. 33.
    Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines - A survey. Proceedings of The IEEE 84(8), 1090–1123 (1996)CrossRefGoogle Scholar
  34. 34.
    Okun, V., Black, P.E., Yesha, Y.: Comparison of fault classes in specification-based testing. Information and Software Technology 46, 525–533 (2004)CrossRefGoogle Scholar
  35. 35.
    Pretschner, A.: Model-based testing in practice. In: Fitzgerald, J.A., Hayes, I.J., Tarlecki, A. (eds.) FM 2005. LNCS, vol. 3582, pp. 537–541. Springer, Heidelberg (2005)Google Scholar
  36. 36.
    Rayadurgam, S., Heimdahl, M.P.: Generating MC/DC adequate test sequences through model checking. In: SEW 2003. 28th Annual NASA Goddard Software Engineering Workshop (2003)Google Scholar
  37. 37.
    Schimd, J.: Executing ASM specifications with AsmGofer,
  38. 38.
    Tsuchiya, T., Kikuno, T.: On fault classes and error detection capability of specification-based testing. ACM Trans. Softw. Eng. Methodol. 11(1), 58–62 (2002)CrossRefGoogle Scholar
  39. 39.
    Weyuker, E., Goradia, T., Singh, A.: Automatically generating test data from a Boolean specification. IEEE Transactions on Software Engineering 20(5), 353–363 (1994)MATHCrossRefGoogle Scholar
  40. 40.
    Zhu, H., Hall, P., May, J.: Software unit test coverage and adequacy. ACM Computing Surveys 29(4), 366–427 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Angelo Gargantini
    • 1
  1. 1.Department of Management and Information Technology, Università di Bergamo 

Personalised recommendations