Test automation for safety-critical systems: Industrial application and future developments

  • Jan Peleska
Invited Lectures
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1051)


Design, execution and evaluation of tests for safety-critical systems require considerable effort and skill and consume a large part of today's development costs. Due to the growing complexity of control systems, it has to be expected that their trustworthy test will become unmanageable in the future, if only conventional techniques, requiring a high degree of human interaction during the test process, are applied. In this article, we will focus on test automation for reactive real-time systems, with emphasis on Hardware-in-the-Loop tests analyzing the behaviour of combined software and hardware components. To illustrate possible approaches for this test problem, we describe a concept based on specifications written in Real-Time CSP. For the implementation of test generation and evaluation algorithms transition system representations are used, as can be obtained by Formal Systems' FDR tool. An industrial application of the method is presented and used for the evaluation of the benefits of formal methods-based testing in comparison with conventional techniques. Furthermore, we will indicate research topics in this field which are likely to become important for further improvements of the test process. Specifically, the benefits arising from an approach combining formal verification and testing will be discussed. Our presentation aims less at “promoting” a specific solution, but tries to illustrate the basic problems to be tackled with any formal method, when trying to develop test automation concepts to be applied in the context of reactive systems.


CSP FDR reactive systems refinement test driver test generation test monitors test oracles 


  1. 1.
    K. R. Apt and E.-R. Olderog. Verification of Sequential and Concurrent Programs. Springer-Verlag, Berlin Heidelberg New York (1991).Google Scholar
  2. 2.
    E. Brinksma: A theory for the derivation of tests. In P. H. J. van Eijk-C. A. Vissers and M. Diaz (Eds.): The Formal Description Technique LOTOS. Elsevire Science Publishers B. V. (North-Holland), (1989), 235–247.Google Scholar
  3. 3.
    RTCA DO178B: Development considerations in airborne computer systems. (1993).Google Scholar
  4. 4.
    J. Davies: Specification and Proof in Real-Time CSP. Cambridge University Press (1993).Google Scholar
  5. 5.
    ELPRO LET GmbH: Programmablaufplan — Bahnübergang. ELPRO LET GmbH (1994).Google Scholar
  6. 6.
    Formal Systems Ltd.: Failures Divergence Refinement. User Manual and Tutorial Version 1.4. Formal Systems (Europe) Ltd (1994).Google Scholar
  7. 7.
    M.-C. Gaudel: Testing can be formal, too. In P. D. Mosses, M. Nielsen and M. I. Schwartzbach (Eds.): Proceedings of TAPSOFT '95: Theory and Practice of Software Development. Aarhus, Denmark, May 1995, Springer (1995).Google Scholar
  8. 8.
    M. C. Hennessy: Algebraic Theory of Processes. MIT Press (1988).Google Scholar
  9. 9.
    C.A.R. Hoare. Communicating sequential processes. Prentice-Hall International, Englewood Cliffs NJ (1985).Google Scholar
  10. 10.
    H. M. Hörcher and J. Peleska: The Role of Formal Specifications in Software Test. Tutorial, held at the FME '94.Google Scholar
  11. 11.
    H. M. Hörcher: Improving Software Tests using Z Specifications. To appear in J. P. Bowen and M. G. Hinchey (Eds.): ZUM '95: 9th International Conference of Z Users, LNCS, Springer (1995).Google Scholar
  12. 12.
    E. Mikk: Compilation of Z Specifications into C for Automatic Test Result Evaluation. To appear in J. P. Bowen and M. G. Hinchey (Eds.): ZUM '95: 9th International Conference of Z Users, LNCS, Springer (1995).Google Scholar
  13. 13.
    R. Milner: Communication and Concurrency. Prentice-Hall International, Englewood Cliffs NJ (1989).Google Scholar
  14. 14.
    M. Müllerburg: Systematic Testing: a Means for Validating Reactive Systems. In EuroSTAR'94: Proceedings of the 2nd European Intern. Conf. on Software Testing, Analysis & Review. British Computer Society, (1994).Google Scholar
  15. 15.
    G. J. Myers: The Art of Software-Testing. John Wiley & Sons, New York (1979).Google Scholar
  16. 16.
    J. Peleska: Trustworthy Tests for Reactive Systems — Automation of Real-Time Testing. In preparation, JP Software-Consulting (1996).Google Scholar
  17. 17.
    J. Peleska and M. Siegel: From Testing Theory to Test Driver Implementation. To appear in Proceedings of the Formal Methods Europe Conference FME '96, LNCS, Springer-Verlag, (1996).Google Scholar
  18. 18.
    J. Peleska: Testautomatisierung für diskrete Steuerungen, Anwendung: Bahnübergangssteuerung — Abschlußbericht Phase 1. Technical Report 06/95, JP Software-Consulting (1995).Google Scholar
  19. 19.
    J. Peleska: Formal Methods and the Development of Dependable Systems. Technical Report 07/95, JP Software-Consulting (1995).Google Scholar
  20. 20.
    D. J. Richardson, T. O. O'Malley and C. Tittle Moore: Approaches to Specification-Based Testing. In ACM SIGSoft 89: Third Symposium on Software Testing, Analysis and Verification, December (1989).Google Scholar
  21. 21.
    D. J. Richardson, S. Leif Aha and T. O. O'Malley: Specification-based Test Oracles for Reactive Systems. In Proceedings of the 14th International Conference on Software Engineering, Melborne, Australia, May (1992).Google Scholar
  22. 22.
    E. Weyuker, T. Goradia and A. Singh: Automatically Generating Test Data from a Boolean Specification. IEEE Transactions on Software Engineering Vol. 20, N0. 5, (1994).Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1996

Authors and Affiliations

  • Jan Peleska
    • 1
    • 2
  1. 1.JP Software-ConsultingGermany
  2. 2.Universität BremenBremenGermany

Personalised recommendations