Skip to main content
SpringerLink
Log in

Industrial-Strength Model-Based Testing of Safety-Critical Systems

  • Conference paper
  • First Online:
FM 2016: Formal Methods (FM 2016)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 9995))

Included in the following conference series:

Abstract

In this article we present an industrial-strength approach to automated model-based testing. This approach is applied by Verified Systems International GmbH in safety-critical verification and validation projects in the avionic, railway, and automotive domains. The SysML modelling formalism is used for creating test models. Associating SysML with a formal behavioural semantics allows for full automation of the whole work flow, as soon as the model including SysML requirements tracing information has been elaborated. The presentation highlights how certain aspects of formal methods are key enablers for achieving the degree of automation that is needed for effectively testing today’s safety critical systems with acceptable effort and the degree of comprehensiveness required by the applicable standards. It is also explained which requirements from the industry and from certification authorities have to be considered when designing test automation tools fit for integration into the verification and validation work flow set up for complex system developments. From the collection of scientific challenges the following questions are addressed. (1) What is the formal equivalent to traceable requirements and associated test cases? (2) How can requirements based, property-based, and model-based testing be effectively automated? (3) Which test strategies provide guaranteed test strength, independent on the syntactic representation of the model?

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://en.wikipedia.org/wiki/Model-based_testing, 2016-07-11.

  2. 2.

    If \(\psi _1\) is stuttering invariant, we have \(\psi _1' = \psi _1\).

  3. 3.

    Note that this simple condition only applies for deterministic state machines; the encoding is more complex for the nondeterministic case.

  4. 4.

    In [30], a finer distinction between fault models, failure models, and defect models is made. Our approach described in this paper is focused on failure models.

References

  1. Anand, S., Burke, E.K., Chen, T.Y., Clark, J.A., Cohen, M.B., Grieskamp, W., Harman, M., Harrold, M.J., McMinn, P.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86(8), 1978–2001 (2013)

    Article  Google Scholar 

  2. Baier, C., Katoen, J.: Principles of Model Checking. MIT Press, Cambridge (2008)

    MATH  Google Scholar 

  3. Biere, A., Heljanko, K., Junttila, T., Latvala, T., Schuppan, V.: Linear encodings of bounded LTL model checking. Logical Methods Comput. Sci. 2(5), 1–64 (2006). arXiv:cs/0611029

    Article  MathSciNet  MATH  Google Scholar 

  4. von Bochmann, G., Das, A., Dssouli, R., Dubuc, M., Ghedamsi, A., Luo, G.: Fault models in testing. In: Kroon, J., Heijink, R.J., Brinksma, E. (eds.) Proceedings of the IFIP TC6/WG6.1 Fourth International Workshop on Protocol Test Systems IV, 15–17 October 1991, Leidschendam, The Netherlands, pp. 17–30. North-Holland (1991). IFIP Transactions, vol. C-3

    Google Scholar 

  5. CENELEC: EN 50128: 2011 Railway applications - Communication, signalling and processing systems - Software for railway control and protection systems (2011)

    Google Scholar 

  6. Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. SE 4(3), 178–186 (1978)

    Article  MATH  Google Scholar 

  7. Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. The MIT Press, Cambridge (1999)

    Google Scholar 

  8. Hennessy, M.: Algebraic Theory of Processes. MIT Press, Cambridge (1988)

    MATH  Google Scholar 

  9. Hierons, R.M.: Testing from a nondeterministic finite state machine using adaptive state counting. IEEE Trans. Comput. 53(10), 1330–1342 (2004). http://doi.ieeecomputersociety.org/10.1109/TC.2004.85

    Article  Google Scholar 

  10. Huang, W., Peleska, J.: Complete model-based equivalence class testing for nondeterministic systems. Formal Aspects of Computing Under review

    Google Scholar 

  11. Huang, W., Peleska, J.: Complete model-based equivalence class testing. STTT 18(3), 265–283 (2016). http://dx.doi.org/10.1007/s10009-014-0356-8

    Article  Google Scholar 

  12. Huang, W.l., Peleska, J., Schulze, U.: Test automation support. Technical report D34.1, COMPASS Comprehensive Modelling for Advanced Systems of Systems (2013). http://www.compass-research.eu/deliverables.html

  13. Hübner, F., Huang, W., Peleska, J.: Experimental evaluation of a novel equivalence class partition testing strategy. In: Blanchette, J.C., Kosmatov, N. (eds.) TAP 2015. LNCS, vol. 9154, pp. 155–172. Springer, Heidelberg (2015). doi:10.1007/978-3-319-21215-9_10

    Chapter  Google Scholar 

  14. ISO, DIS 26262–4: Road vehicles - functional safety - part 4: Product development: system level. Technical report, International Organization for Standardization (2009)

    Google Scholar 

  15. ISO, IEC, IEEE DIS 29119–4.2: Software and systems engineering - software testing - part: 4 test techniques, February 2014

    Google Scholar 

  16. Luo, G., von Bochmann, G., Petrenko, A.: Test selection based on communicating nondeterministic finite-state machines using a generalized WP-method. IEEE Trans. Softw. Eng. 20(2), 149–162 (1994). http://doi.ieeecomputersociety.org/10.1109/32.265636

    Article  Google Scholar 

  17. Morell, L.J.: A theory of fault-based testing. IEEE Trans. Softw. Eng. 16(8), 844–857 (1990). http://dx.doi.org/10.1109/32.57623

    Article  Google Scholar 

  18. Object Management Group: Object Constraint Language, Version 2.4. Technical report, Object Management Group (2014). http://www.omg.org/spec/OCL/2.4/

  19. Object Management Group: OMG Systems Modeling Language (OMG SysML), Version 1.4. Technical report, Object Management Group (2015). http://www.omg.org/spec/SysML/1.4

  20. Peleska, J.: Formal methods and the development of dependable systems. No. 9612, Christian-Albrechts-Universität Kiel, Institut fr Informatik und Praktische Mathematik , Habilitationsschrift, December 1996

    Google Scholar 

  21. Peleska, J.: Industrial-strength model-based testing-state of the art and current challenges. In: Petrenko, A.K., Schlingloff, H. (eds.) Proceedings Eighth Workshop on Model-Based Testing. Electronic Proceedings in Theoretical Computer Science, 17th March 2013, Rome, Italy, vol. 111, pp. 3–28. Open Publishing Association (2013)

    Google Scholar 

  22. Peleska, J., Honisch, A., Lapschies, F., Löding, H., Schmid, H., Smuda, P., Vorobev, E., Zahlten, C.: A real-world benchmark model for testing concurrent real-time systems in the automotive domain. In: Wolff, B., Zaïdi, F. (eds.) ICTSS 2011. LNCS, vol. 7019, pp. 146–161. Springer, Heidelberg (2011). doi:10.1007/978-3-642-24580-0_11

    Chapter  Google Scholar 

  23. Peleska, J., Huang, W.: Model-based testing strategies and their (in)dependence on syntactic model representations. In: Beek, M.H., Gnesi, S., Knapp, A. (eds.) FMICS/AVoCS -2016. LNCS, vol. 9933, pp. 3–21. Springer, Heidelberg (2016). doi:10.1007/978-3-319-45943-1_1

    Chapter  Google Scholar 

  24. Peleska, J., Huang, W., Hübner, F.: A novel approach to HW/SW integration testing of route-based interlocking system controllers. In: Lecomte, T., Pinger, R., Romanovsky, A. (eds.) RSSRail 2016. LNCS, vol. 9707, pp. 32–49. Springer, Heidelberg (2016). doi:10.1007/978-3-319-33951-1_3

    Chapter  Google Scholar 

  25. Peleska, J., Vorobev, E., Lapschies, F.: Automated test case generation with SMT-solving and abstract interpretation. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 298–312. Springer, Heidelberg (2011). doi:10.1007/978-3-642-20398-5_22

    Chapter  Google Scholar 

  26. Petrenko, A., Yevtushenko, N.: Adaptive testing of deterministic implementations specified by nondeterministic FSMs. In: Wolff, B., Zaïdi, F. (eds.) ICTSS 2011. LNCS, vol. 7019, pp. 162–178. Springer, Heidelberg (2011). doi:10.1007/978-3-642-24580-0_12

    Chapter  Google Scholar 

  27. Petrenko, A., Yevtushenko, N., Bochmann, G.v.: Fault models for testing in context. In: Gotzhein, R., Bredereke, J. (eds.) Formal Description Techniques IX - Theory, Application and Tools, pp. 163–177. Chapman & Hall (1996)

    Google Scholar 

  28. Petrenko, A., Yevtushenko, N., Bochmann, G.V.: Testing deterministic implementations from nondeterministic FSM specifications. In: IFIP TC6 9th International Workshop on Testing of Communicating Systems, pp. 125–141. Chapman and Hall (1996)

    Google Scholar 

  29. Petrenko, A., Simao, A., Maldonado, J.C.: Model-based testing of software and systems: recent advances and challenges. Int. J. Softw. Tools Technol. Transf. 14(4), 383–386 (2012). http://dx.doi.org/10.1007/s10009-012-0240-3

    Article  Google Scholar 

  30. Pretschner, A.: Defect-based testing. In: Irlbeck, M., Peled, D.A., Pretschner, A. (eds.) Dependable Software Systems Engineering, NATO Science for Peace and Security Series, D: Information and Communication Security, vol. 40, pp. 224–245. IOS Press (2015). http://dx.doi.org/10.3233/978-1-61499-495-4-224

  31. Sistla, A.P.: Safety, liveness and fairness in temporal logic. Formal Asp. Comput. 6(5), 495–512 (1994). http://dx.doi.org/10.1007/BF01211865

    Article  MATH  Google Scholar 

  32. Soden, M., Eichler, H.: Temporal extensions of OCL revisited. In: Paige, R.F., Hartman, A., Rensink, A. (eds.) ECMDA-FA 2009. LNCS, vol. 5562, pp. 190–205. Springer, Heidelberg (2009). doi:10.1007/978-3-642-02674-4_14

    Chapter  Google Scholar 

  33. Tretmans, J.: Conformance testing with labelled transition systems: implementation relations and test generation. Comput. Netw. ISDN Syst. 29(1), 49–79 (1996)

    Article  Google Scholar 

  34. Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 (2012). http://dx.doi.org/10.1002/stvr.456

    Article  Google Scholar 

  35. Vasilevskii, M.P.: Failure diagnosis of automata. Kibernetika (Transl.) 4, 98–108 (1973)

    MathSciNet  Google Scholar 

  36. WG-71, R.S.E.: Software Considerations in Airborne Systems and Equipment Certification. Technical report RTCA/DO-178C, RTCA Inc., 1140 Connecticut Avenue, N.W., Suite 1020, Washington, D.C. 20036, December 2011

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the members of the FM 2016 program committee for the invitation to present this paper.

We are also very grateful to our collaborators at the University of Bremen and Verified Systems International who contributed to the development of RT-Tester’s MBT component; in particular we would like to thank Felix Hübner, Uwe Schulze, and Jörg Brauer.

The work presented in this paper has been elaborated within project ITTCPS – Implementable Testing Theory for Cyber-physical Systems (see http://www.informatik.uni-bremen.de/agbs/projects/ittcps/index.html) which has been granted by the University of Bremen in the context of the German Universities Excellence Initiative (see http://en.wikipedia.org/wiki/German_Universities_Excellence_Initiative).

Author information

Authors and Affiliations

  1. Verified Systems International GmbH, Bremen, Germany

    Jan Peleska

  2. Department of Mathematics and Computer Science, University of Bremen, Bremen, Germany

    Jan Peleska & Wen-ling Huang

Authors
  1. Jan Peleska
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wen-ling Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jan Peleska .

Editor information

Editors and Affiliations

  1. Newcastle University, Newcastle upon Tyne, United Kingdom

    John Fitzgerald

  2. US Naval Research Laboratory, Washington, District of Columbia, USA

    Constance Heitmeyer

  3. ISTI-CNR, Pisa, Italy

    Stefania Gnesi

  4. University of Cyprus, Nicosia, Cyprus

    Anna Philippou

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Peleska, J., Huang, Wl. (2016). Industrial-Strength Model-Based Testing of Safety-Critical Systems. In: Fitzgerald, J., Heitmeyer, C., Gnesi, S., Philippou, A. (eds) FM 2016: Formal Methods. FM 2016. Lecture Notes in Computer Science(), vol 9995. Springer, Cham. https://doi.org/10.1007/978-3-319-48989-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-48989-6_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-48988-9

  • Online ISBN: 978-3-319-48989-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation