Formal Methods in System Design

, Volume 34, Issue 3, pp 238–304 | Cite as

Conformance testing for real-time systems

  • Moez Krichen
  • Stavros Tripakis


We propose a new framework for black-box conformance testing of real-time systems. The framework is based on the model of partially-observable, non-deterministic timed automata. We argue that partial observability and non-determinism are essential features for ease of modeling, expressiveness and implementability. The framework allows the user to define, through appropriate modeling, assumptions on the environment of the system under test (SUT) as well as on the interface between the tester and the SUT. We consider two types of tests: analog-clock tests and digital-clock tests. Our algorithm for generating analog-clock tests is based on an on-the-fly determinization of the specification automaton during the execution of the test, which in turn relies on reachability computations. The latter can sometimes be costly, thus problematic, since the tester must quickly react to the actions of the system under test. Therefore, we provide techniques which allow analog-clock testers to be represented as deterministic timed automata, thus minimizing the reaction time to a simple state jump. We also provide algorithms for static or on-the-fly generation of digital-clock tests. These tests measure time only with finite-precision digital clocks, another essential condition for implementability. We also propose a technique for location, edge and state coverage of the specification, by reducing the problem to covering a symbolic reachability graph. This avoids having to generate too many tests. We report on a prototype tool called \(\mathsf{TTG}\) and two case studies: a lighting device and the Bounded Retransmission Protocol. Experimental results obtained by applying \(\mathsf{TTG}\) on the Bounded Retransmission Protocol show that only a few tests suffice to cover thousands of reachable symbolic states in the specification.


Conformance testing Test generation Coverage Partial observability On-the-fly algorithms Real-time systems Timed automata Specification and verification 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alur R, Dill D (1994) A theory of timed automata. Theor Comput Sci 126:183–235 zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Alur R, Fix L, Henzinger T (1994) A determinizable class of timed automata. In: CAV’94. LNCS, vol 818. Springer, Berlin Google Scholar
  3. 3.
    Belinfante A, Feenstra J, de Vries RG, Tretmans J, Goga N, Feijs L, Mauw S, Heerink L (1999) Formal test automation: a simple experiment. In: 12th international workshop on testing of communicating systems. Kluwer, Dordrecht Google Scholar
  4. 4.
    Bengtsson J, Yi W (2003) On clock difference constraints and termination in reachability analysis of timed automata. In: ICFEM’03. LNCS, vol 2885. Springer, Berlin Google Scholar
  5. 5.
    Bensalem S, Bozga M, Krichen M, Tripakis S (2005) Testing conformance of real-time applications by automatic generation of observers. In: 4th international workshop on runtime verification (RV’04). ENTCS, vol 113. Elsevier, Amsterdam, pp 23–43 Google Scholar
  6. 6.
    Berard B, Petit A, Diekert V, Gastin P (1998) Characterization of the expressive power of silent transitions in timed automata. Fund Inform 36(2–3):145–182 zbMATHMathSciNetGoogle Scholar
  7. 7.
    Berthomieu B, Menasche M (1983) An enumerative approach for analyzing time Petri nets. IFIP Congr Ser 9:41–46 Google Scholar
  8. 8.
    Blom J, Hessel A, Jonsson B, Pettersson P (2004) Specifying and generating test cases using observer automata. In: Proceedings of formal approaches to software testing, pp 125–139 Google Scholar
  9. 9.
    Bornot S, Sifakis J, Tripakis S (1998) Modeling urgency in timed systems. In: Compositionality. LNCS, vol 1536. Springer, Berlin Google Scholar
  10. 10.
    Bouyer P (2004) Forward analysis of updatable timed automata. Formal Methods in System Design 24(3):281–320 zbMATHCrossRefGoogle Scholar
  11. 11.
    Bouyer P, Chevalier F, D’Souza D (2005) Fault diagnosis using timed automata. In: FoSSaCS’05. LNCS, vol 3441. Springer, Berlin, pp 219–233 Google Scholar
  12. 12.
    Bozga M, Fernandez JC, Ghirvu L, Graf S, Krimm JP, Mounier L (2000) IF: a validation environment for timed asynchronous systems. In: Proc CAV’00. LNCS, vol 1855. Springer, Berlin Google Scholar
  13. 13.
    Bozga M, Graf S, Kerbrat A, Mounier L, Ober I, Vincent D (2000) SDL for real-time: what is missing? In: Proceedings of SAM’00: 2nd workshop on SDL and MSC, Grenoble, France, pp 108–122. IMAG, June 2000 Google Scholar
  14. 14.
    Braberman V, Felder M, Marre M (1997) Testing timing behavior of real-time software. In: International software quality week Google Scholar
  15. 15.
    Brat G, Giannakopoulou D, Goldberg A, Havelund K, Lowry M, Pasareanu C, Venet A, Visser W (2003) Experimental evaluation of V&V tools on martian rover software. In: SEI software model checking workshop Google Scholar
  16. 16.
    Brinksma E, Tretmans J (2001) Testing transition systems: an annotated bibliography. In: MOVEP 2000. LNCS, vol 2067. Springer, Berlin Google Scholar
  17. 17.
    Briones L, Brinksma E (2004) A test generation framework for quiescent real-time systems. In: FATES’04. LNCS, vol 3395. Springer, Berlin Google Scholar
  18. 18.
    Cardell-Oliver R (2002) Conformance test experiments for distributed real-time systems. In: ISSTA’02. ACM, New York Google Scholar
  19. 19.
    Chow TS (1978) Testing software design modeled by finite-state machines. IEEE Trans Softw Eng 4(1) Google Scholar
  20. 20.
    Clarke D, Jéron T, Rusu V, Zinovieva E (2002) STG: a symbolic test generation tool. In: TACAS’02. LNCS, vol 2280. Springer, Berlin Google Scholar
  21. 21.
    Clarke D, Lee I (1997) Automatic generation of tests for timing constraints from requirements. In: 3rd workshop on object-oriented real-time dependable systems (WORDS’97) Google Scholar
  22. 22.
    Daws C, Olivero A, Tripakis S, Yovine S (1996) The tool Kronos. In: Hybrid systems III, verification and control. LNCS, vol 1066. Springer, Berlin, pp 208–219 Google Scholar
  23. 23.
    Daws C, Tripakis S (1998) Model checking of real-time reachability properties using abstractions. In: Tools and algorithms for the construction and analysis of systems ’98, Lisbon, Portugal. LNCS, vol 1384. Springer, Berlin CrossRefGoogle Scholar
  24. 24.
    Dill D (1989) Timing assumptions and verification of finite-state concurrent systems. In: Sifakis J (ed) Automatic verification methods for finite state systems. LNCS, vol 407. Springer, Berlin, pp 197–212 Google Scholar
  25. 25.
    En-Nouaary A, Dssouli R, Khendek F, Elqortobi A (1998) Timed test cases generation based on state characterization technique. In: RTSS’98. IEEE, New York Google Scholar
  26. 26.
    Fernandez JC, Jard C, Jéron T, Viho G (1996) Using on-the-fly verification techniques for the generation of test suites. In: CAV’96. LNCS, vol 1102. Springer, Berlin Google Scholar
  27. 27.
    Groote JF, van de Pol J (1996) A bounded retransmission protocol for large data packets. In: Algebraic methodology and software technology, pp 536–550 Google Scholar
  28. 28.
    Henzinger T, Manna Z, Pnueli A (1992) What good are digital clocks? In: ICALP’92. LNCS, vol 623. Springer, Berlin Google Scholar
  29. 29.
    Hessel A, Larsen K, Nielsen B, Pettersson P, Skou A (2003) Time-optimal real-time test case generation using UPPAAL. In: FATES’03 Google Scholar
  30. 30.
    Higashino T, Nakata A, Taniguchi K, Cavalli A (1999) Generating test cases for a timed I/O automaton model. In: IFIP international workshop on testing of Communicating Systems. Kluwer, Dordrecht Google Scholar
  31. 31.
    ISO/IEC (1992) Open systems interconnection conformance testing methodology and framework—part 1: general concept; part 2: abstract test suite specification; part 3: the tree and tabular combined notation (TTCN). Technical Report 9646, International Organization for Standardization—Information Processing Systems—Open Systems Interconnection, Genève Google Scholar
  32. 32.
    Jard C, Jéron T, Morel P (2000) Verification of test suites. In: TestCom 2000 Google Scholar
  33. 33.
    Khoumsi A, Jéron T, Marchand H (2003) Test cases generation for nondeterministic real-time systems. In: FATES’03 Google Scholar
  34. 34.
    Krichen M, Tripakis S (2004) Black-box conformance testing for real-time systems. In: 11th international spin workshop on model checking of software (SPIN’04). LNCS, vol 2989. Springer, Berlin Google Scholar
  35. 35.
    Krichen M, Tripakis S (2004) Real-time testing with timed automata testers and coverage criteria. In: Formal techniques, modelling and analysis of timed and fault tolerant systems (FORMATS-FTRTFT’04). LNCS, vol 3253. Springer, Berlin Google Scholar
  36. 36.
    Krichen M, Tripakis S (2005) An expressive and implementable formal framework for testing real-time systems. In: The 17th IFIP international conference on testing of communicating systems (TestCom’05). LNCS, vol 3502. Springer, Berlin Google Scholar
  37. 37.
    Krichen M, Tripakis S (2005) State identification problems for timed automata. In: The 17th IFIP international conference on testing of communicating systems (TestCom’05). LNCS, vol 3502. Springer, Berlin Google Scholar
  38. 38.
    Larsen K, Mikucionis M, Nielsen B (2004) Online testing of real-time systems using uppaal. In: FATES’04. LNCS, vol 3395. Springer, Berlin Google Scholar
  39. 39.
    Larsen K, Mikucionis M, Nielsen B, Skou A (2005) Testing real-time embedded software using UPPAAL-TRON: an industrial case study. In: 5th ACM international conference on embedded software. ACM, New York, pp 299–306 CrossRefGoogle Scholar
  40. 40.
    Lee D, Yannakakis M (1996) Principles and methods of testing finite state machines—a survey. Proc IEEE 84:1090–1126 CrossRefGoogle Scholar
  41. 41.
    Myers GJ (1979) The art of software testing. Wiley, New York Google Scholar
  42. 42.
    Nielsen B, Skou A (2001) Automated test generation from timed automata. In: TACAS’01. LNCS, vol 2031. Springer, Berlin Google Scholar
  43. 43.
    Peleska J (2002) Formal methods for test automation—hard real-time testing of controllers for the airbus aircraft family. In: IDPT’02 Google Scholar
  44. 44.
    Puri A (2000) Dynamical properties of timed automata. Discrete Event Dyn Syst 10(1–2):87–113 zbMATHCrossRefMathSciNetGoogle Scholar
  45. 45.
    Sifakis J, Yovine S (1996) Compositional specification of timed systems. In: 13th Annual symposium on theoretical aspects of computer science, STACS’96. LNCS, vol 1046. Springer, Berlin Google Scholar
  46. 46.
    Springintveld J, Vaandrager F, D’Argenio P (2001) Testing timed automata. Theor Comput Sci 254 Google Scholar
  47. 47.
    Tretmans J (1999) Testing concurrent systems: a formal approach. In: CONCUR’99. LNCS, vol 1664. Springer, Berlin Google Scholar
  48. 48.
    Tretmans J (2002) Testing techniques. Lecture notes. University of Twente, The Netherlands Google Scholar
  49. 49.
    Tripakis S (2002) Fault diagnosis for timed automata. In: Formal techniques in real time and fault tolerant systems (FTRTFT’02). LNCS, vol 2469. Springer, Berlin CrossRefGoogle Scholar
  50. 50.
    Tripakis S (2004) Folk theorems on the determinization and minimization of timed automata. In: Formal modeling and analysis of timed systems (FORMATS’03). LNCS, vol 2791. Springer, Berlin Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Verimag LaboratoryCentre EquationGièresFrance

Personalised recommendations