Advertisement

Model-based testing for real

The inhouse card case study
  • A. Pretschner
  • O. Slotosch
  • E. Aiglstorfer
  • S. Kriebel
Special section on formal methods for industrial critical systems

Abstract

Model-based testing relies on abstract behavior models for test case generation. These models are abstractions, i.e., simplifications. For deterministic reactive systems, test cases are sequences of input and expected output. To bridge the different levels of abstraction, input must be concretized before being applied to the system under test. The system’s output must then be abstracted before being compared to the output of the model.

The concepts are discussed along the lines of a feasibility study, an inhouse smart card case study. We describe the modeling concepts of the CASE tool AutoFocus and an approach to model-based test case generation that is based on symbolic execution with Constraint Logic Programming.

Different search strategies and algorithms for test case generation are discussed. Besides validating the model itself, generated test cases were used to verify the actual hardware with respect to these traces.

Keywords

Test case generation Model checking Symbolic execution Behavior models 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Ammann P, Black P, Majurski W (1998) Using model checking to generate tests from specifications. In: Proceedings of the 2nd IEEE international conference on formal engineering methods, Brisbane, Queensland, Australia, 9–11 December 1998, pp 46–54 Google Scholar
  2. 2.
    Binder R (2001) Testing object-oriented systems: models, patterns, and tools. Addison-Wesley, Reading, MA Google Scholar
  3. 3.
    Bourhfir C, Dssouli R, Aboulhamid E (1996) Automatic test generation for EFSM-based systems. Technical Report IRO 1043, University of Montreal, August 1996 Google Scholar
  4. 4.
    Brooks F (1986) No silver bullet. In: Proceedings of the 10th IFIP world computing conference, Dublin, Ireland, 1–5 September 1986, pp 1069–1076 Google Scholar
  5. 5.
    Bultan T (1998) Automated symbolic analysis of reactive systems. PhD thesis, University of Maryland, College Park, MD Google Scholar
  6. 6.
    Burton S, Clark J, McDermid J (2001) Automatic generation of tests from Statechart specifications. In: Proceedings of the conference on formal approaches to testing of software, Aalborg, Denmark, 25 August 2001, pp 31–46 Google Scholar
  7. 7.
    Chow T (1978) Testing software design modeled by finite-state machines. IEEE Trans Softw Eng SE-4(3):178–187 Google Scholar
  8. 8.
    Ciarlini A, Frühwirth T (1999) Using Constraint Logic Programming for software validation. In: Proceedings of the 5th workshop on the German-Brazilian Bilateral Programme for Scientific and Technological Cooperation, Königswinter, Germany, March 1999 Google Scholar
  9. 9.
    Claessen K, Hughes J (2000) QuickCheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of the 5th ACM SIGPLAN international conference on functional programming, Montreal, 18–21 September 2000, pp 268–279 Google Scholar
  10. 10.
    Clarke L (1976) A system to generate test data and symbolically execute programs. IEEE Trans Softw Eng SE-2(3):215–222 Google Scholar
  11. 11.
    Cui B, Dong Y, Du X, Kumar NK, Ramakrishnan C, Ramakrishnan I, Roychoudhury A, Smolka S, Warren D (1998) Logic programming and model checking. Lecture notes in computer science, vol 1490. Springer, Berlin Heidelberg New York, pp 1–20 Google Scholar
  12. 12.
    Delzanno G, Podelski A (1999) Model checking in CLP. In: Proceedings of the 5th international conference on tools and algorithms for construction and analysis of systems, Amsterdam, 22–28 March 1999, pp 223–239 Google Scholar
  13. 13.
    Denney R (1991) Test-case generation from Prolog-based specifications. IEEE Softw 8(2):49–57 CrossRefGoogle Scholar
  14. 14.
    De Vries R, Tretmans J, Belinfante A, Feenstra J, Feijs L, Mauw S, Goga N, Heerink L, de Heer A (2000) Côte de Resyste in Progress. In: Proceedings of Progress 2000 – workshop on embedded systems, Utrecht, The Netherlands, October 2000, pp 141–148 Google Scholar
  15. 15.
    Du Bousquet L, Ouabdesselam F, Parissis I, Richier J-L, Zuanon N (2000) Specification-based testing of synchronous software. In: Proceedings of the 5th international workshop on formal methods for industrial critical systems, Berlin, 3–4 April 2000, pp 123–140 Google Scholar
  16. 16.
    Du Bousquet L, Zuanon N (1999) An overview of Lutess, a specification-based tool for testing synchronous software. In: Proceedings of the 14th IEEE international conference on automated SW engineering, Cocoa Beach, FL, 12–15 October 1999, pp 208–215 Google Scholar
  17. 17.
    Duran J, Ntafos S (1984) An evaluation of random testing. IEEE Trans Softw Eng SE-10(4):438–444 Google Scholar
  18. 18.
    Edelkamp S, Lluch-Lafuente A, Leue S (2001) Directed Explicit Model Checking with HSF-SPIN. In: Proceedings of the 8th international SPIN workshop on model checking software, Toronto, 19–20 May 2001, pp 57–79 Google Scholar
  19. 19.
    Fernandez J-C, Jard C, Jéron T, Viho C (1996) Using on-the-fly verification techniques for the generation of test suites. In: Proceedings of the 8th international conference on computer-aided verification, New Brunswick, NJ, 31 July–3 August 1996, pp 348–359 Google Scholar
  20. 20.
    Frankl P, Weyuker E (1998) An applicable family of data flow testing criteria. IEEE Trans Softw Eng 14(10):1483–1498 CrossRefGoogle Scholar
  21. 21.
    Fribourg L (1999) Constraint logic programming applied to model checking. In: Proceedings of the 9th international workshop on logic-based program synthesis and transformation (LOPSTR’99), Venice, 22–24 September 1999. Lecture notes in computer science, vol 1817. Springer, Berlin Heidelberg New York, pp 30–41 Google Scholar
  22. 22.
    Frühwirth T (1998) Theory and practice of constraint handling rules. J Logic Program 37(1–3):95–138 Google Scholar
  23. 23.
    Goodenough J, Gerhart S (1975) Toward a theory of test data selection. IEEE Trans Softw Eng SE-1(2):156–173 Google Scholar
  24. 24.
    Groce A, Visser W (2002) Model checking Java programs using structural heuristics. In: Proceedings of the international symposium on software testing and analysis, Rome, 22–24 July 2002, pp 12–21 Google Scholar
  25. 25.
    Gutjahr W (1999) Partition testing versus random testing: the influence of uncertainty. IEEE Trans Softw Eng 25(5):661–674 CrossRefGoogle Scholar
  26. 26.
    Hahn G, Philipps J, Pretschner A, Stauner T (2003) Prototype-based tests for hybrid reactive systems. In: Proceedings of RSP’03, San Diego, 9–11 June 2003, pp 78–86 Google Scholar
  27. 27.
    Hamlet D, Taylor R (1990) Partition test does not inspire confidence. IEEE Trans Softw Eng 16(12):1402–1411 MathSciNetCrossRefGoogle Scholar
  28. 28.
    Howden W (1975) Methodology for the generation of program test data. IEEE Trans Comput C-24(5):554–560 Google Scholar
  29. 29.
    Howden W (1977) Symbolic testing and the DISSECT symbolic evaluation system. IEEE Trans Softw Eng SE-3(4):266–278 Google Scholar
  30. 30.
    Howden W (1978) An evaluation of the effectiveness of symbolic testing. Softw Pract Exper 8:381–397 CrossRefGoogle Scholar
  31. 31.
    Huber F, Schätz B, Einert G (1997) Consistent graphical specification of distributed systems. In: Proceedings of the conference on industrial applications and strengthened foundations of formal methods (FME’97), Graz, Austria, 15–19 September 1997. Lecture notes in computer science, vol 1313. Springer, Berlin Heidelberg New York, pp 122–141 Google Scholar
  32. 32.
    International Organization for Standardization (1995) International Standard ISO/IEC 7816: integrated circuit(s) cards with contacts Google Scholar
  33. 33.
    King J (1976) Symbolic execution and program testing. Commun ACM 19(7):385–394 CrossRefGoogle Scholar
  34. 34.
    Koch B, Grabowski J, Hogrefe D, Schmitt M (1998) AutoLink – a tool for automatic test generation from SDL specifications. In: Proceedings of the IEEE international workshop on industrial strength formal specification techniques, Boca Raton, FL, 20–23 October 1998, pp 114–127 Google Scholar
  35. 35.
    Legeard B, Peureux F (2001) Génération de séquences de tests à partir d’une spécification B en PLC ensembliste. In: Proceedings of Approches Formelles dans l’Assistance au Développement de Logiciels, Nancy, France, June 2001, pp 113–130 Google Scholar
  36. 36.
    Lötzbeyer H, Pretschner A (2000) AutoFocus on Constraint Logic Programming. In: Proceedings of (Constraint) Logic Programming and Software Engineering, London, 10 August 2000 Google Scholar
  37. 37.
    Lötzbeyer H, Pretschner A (2000) Testing concurrent reactive systems with Constraint Logic Programming. In: Proceedings of the 2nd workshop on rule-based constraint reasoning and programming, Singapore, 22 September 2000 Google Scholar
  38. 38.
    Marre B, Arnould A (2000) Test sequence generation from Lustre descriptions: GATEL. In: Proceedings of the 15th IEEE international conference on automated software engineering (ASE’00), Grenoble, France, 11–15 September 2000, pp 229-238 Google Scholar
  39. 39.
    Meudec C (2000) ATGen: automatic test data generation using Constraint Logic Programming and Symbolic Execution. In: Proceedings of the 1st international workshop on automated program analysis, testing, and verification, Limerick, Ireland, 4–5 June 2000 Google Scholar
  40. 40.
    Nielsen B (2000) Specification and test of real-time systems. PhD thesis, Department of Computer Science, Aalborg University, Aalborg, Denmark Google Scholar
  41. 41.
    Ntafos S (1988) A comparison of some structural testing strategies. IEEE Trans Softw Eng 14(6):868–874 CrossRefGoogle Scholar
  42. 42.
    Peleska J, Siegel M (1997) Test automation of safety-critical reactive systems. S Afric Comput J 19:53–77 Google Scholar
  43. 43.
    Philipps J, Pretschner A, Slotosch O, Aiglstorfer E, Kriebel S, Scholl K (2003) Model-based test case generation for smart cards. In: Proceedings of FMICS’03, Trondheim, Norway, 5–7 June 2003, pp 168–182 Google Scholar
  44. 44.
    Prenninger W, Pretschner A, Wagner S (2003) MOST NetworkMaster – AutoFocus model. Internal Study, BMW AG and TU München, Munich, Germany Google Scholar
  45. 45.
    Prenninger W, Pretschner A, Wagner S (2003) MOST NetworkMaster – generation of test harnesses. Internal Study, BMW AG and TU München, Munich, Germany Google Scholar
  46. 46.
    Pretschner A (2001) Classical search strategies for test case generation with Constraint Logic Programming. In: Proceedings of the workshop on formal approaches to testing of software, Aalborg, Denmark, August 2001, pp 47–60 Google Scholar
  47. 47.
    Pretschner A (2003) Compositional generation for MC/DC test suites. In: Proceedings of TACoS’03, Warsaw, Poland, 13 April 2003, pp 1–11 Google Scholar
  48. 48.
    Pretschner A, Lötzbeyer H, Philipps J (2001) Model based testing in evolutionary software development. In: Proceedings of the 11th IEEE international workshop on rapid system prototyping, Monterey, CA, 25–27 June 2001, pp 155–160 Google Scholar
  49. 49.
    Pretschner A, Lötzbeyer H, Philipps J (2003) Model based testing in incremental system development. J Sys Softw 70(3):315–329 CrossRefGoogle Scholar
  50. 50.
    Pretschner A, Philipps J (2002) Szenarien modellbasierten Testens. Technical Report TUM-I0205, Institut für Informatik, Technische Universität München, Munich, Germany Google Scholar
  51. 51.
    Pretschner A, Slotosch O, Lötzbeyer H, Aiglstorfer E, Kriebel S (2001) Model based testing for real: the inhouse card case study. In: Proceedings of the 6th international workshop on formal methods for industrial critical systems, Paris, France, 16–17 July 2001, pp 79–94 Google Scholar
  52. 52.
    Prowell S, Trammell C, Linger R, Poore J (1999) Cleanroom software engineering. Addison-Wesley, Reading, MA Google Scholar
  53. 53.
    Ramamoorthy C, Ho S, Chen W (1976) On the automated generation of program test data. IEEE Trans Softw Eng SE-2(4):293–300 Google Scholar
  54. 54.
    Raymond P, Weber D, Nicollin X, Halbwachs N (1998) Automatic testing of reactive systems. In: Proceedings of the 19th IEEE symposium on real-time systems, Madrid, 2–4 December 1998, pp 200–209 Google Scholar
  55. 55.
    Rusu V, du Bousquet L, Jéron T (2000) An approach to symbolic test generation. In: Proceedings of Integrated Formal Methods, Dagstuhl, Germany, 1–3 November 2000, pp 338–357 Google Scholar
  56. 56.
    Sadeghipour S (1998) Testing cyclic software components of reactive systems on the basis of formal specifications. PhD thesis, Department of Informatics, TU Berlin Google Scholar
  57. 57.
    Tracey N (2000) A search-based automated test-data generation framework for safety-critical software. PhD thesis, Department of Computer Science, University of York, UK Google Scholar
  58. 58.
    Tretmans J (1996) Test generation with inputs, outputs and repetitive quiescence. Softw Concepts Tools 17(3):103–120 Google Scholar
  59. 59.
    Ural H (1992) Formal methods for test sequence generation. Comput Commun 15(5):311–325 CrossRefGoogle Scholar
  60. 60.
    Vilkomir S, Bowen J (2001) Formalization of control-flow criteria of software testing. Technical Report SBU-CISM-01-01, South Bank University, London, UK Google Scholar
  61. 61.
    Visser W, Havelund K, Brat G, Park S (2000) Java PathFinder – second generation of a Java model checker. In: Proceedings of the workshop on advances in verification, Chicago, July 2000 Google Scholar
  62. 62.
    Von Bochmann G, Petrenko A (1994) Protocol testing: review of methods and testing for software testing. In: Proceedings of the 1994 international symposium on software testing and analysis, Seattle, 17–19 August 1994, pp 109–124 Google Scholar
  63. 63.
    Warren DS (1992) Memoing for logic programs. Commun ACM 35(3):93–111 CrossRefGoogle Scholar
  64. 64.
    Wegener J (2001) Evolutionärer Test des Zeitverhaltens von Realzeit-Systemen. PhD thesis, Humboldt Universität, Berlin Google Scholar
  65. 65.
    Weyuker E (1986) Axiomatizing software test data adequacy. IEEE Trans Softw Eng SE-12(12):1128–1138 Google Scholar
  66. 66.
    Wimmel G, Lötzbeyer H, Pretschner A, Slotosch O (2000) Specification based test sequence generation with propositional logic. J Softw Test Validat Reliabil 10(4):229–248 CrossRefGoogle Scholar
  67. 67.
    Zhu H, Hall P, May J (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4):366–427CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2004

Authors and Affiliations

  • A. Pretschner
    • 1
  • O. Slotosch
    • 2
  • E. Aiglstorfer
    • 3
  • S. Kriebel
    • 4
  1. 1.Department of Computer ScienceETH ZurichZurichSwitzerland
  2. 2.Validas Model Validation AG, gateGarchingGermany
  3. 3.Giesecke&Devrient GmbHMunichGermany
  4. 4.BMW GroupMunichGermany

Personalised recommendations