Skip to main content

Automated Requirements-Based Testing of Black-Box Reactive Systems

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 12229)

Abstract

We present a new approach to conformance testing of black-box reactive systems. We consider system specifications written as linear temporal logic formulas to generate tests as sequences of input/output pairs: inputs are extracted from the Büchi automata corresponding to the specifications, and outputs are obtained by feeding the inputs to the systems. Conformance is checked by comparing input/output sequences with automata traces to detect violations of the specifications. We consider several criteria for extracting tests and for stopping generation, and we compare them experimentally using both indicators of coverage and error-detection. The results show that our methodology can generate test suites with good system coverage and error-detection capability.

Keywords

  • Automated testing and verification
  • Runtime verification
  • Black-box conformance testing

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-55754-6_9
  • Chapter length: 17 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   69.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-55754-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   89.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.

Notes

  1. 1.

    http://www.syntcomp.org/.

  2. 2.

    Using the command line ltl2tgba -B -f “p0 \(-\) (X G p1 | ! F p1)”.

  3. 3.

    Fired with command line ltl2tgba -MD -f “p0 \(-\) (X G p1 | ! F p1)”.

  4. 4.

    All benchmarks are available at https://gitlab.sagelab.it/sage/benchmarks-tests.

  5. 5.

    http://www.syntcomp.org/.

  6. 6.

    http://cerbero-h2020.eu.

  7. 7.

    http://www.trossenrobotics.com/widowxrobotarm.

  8. 8.

    https://gitlab.sagelab.it/sage/SpecPro.

References

  1. Andrews, J.H., Briand, L.C., Labiche, Y., Namin, A.S.: Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng. 32(8), 608–624 (2006)

    CrossRef  Google Scholar 

  2. Aniculaesei, A., Howar, F., Denecke, P., Rausch, A.: Automated generation of requirements-based test cases for an adaptive cruise control system. In: 2018 IEEE Workshop on Validation, Analysis and Evolution of Software Tests (VST), pp. 11–15. IEEE (2018)

    Google Scholar 

  3. Arcaini, P., Gargantini, A., Riccobene, E.: Online testing of LTL properties for java code. In: Bertacco, V., Legay, A. (eds.) HVC 2013. LNCS, vol. 8244, pp. 95–111. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-03077-7_7

    CrossRef  Google Scholar 

  4. Baier, C., Katoen, J.P.: Principles of Model Checking. MIT press, Cambridge (2008)

    MATH  Google Scholar 

  5. Barbot, B., Basset, N., Dang, T.: Generation of signals under temporal constraints for CPS testing. In: Badger, J.M., Rozier, K.Y. (eds.) NFM 2019. LNCS, vol. 11460, pp. 54–70. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20652-9_4

    CrossRef  Google Scholar 

  6. Bartocci, E., Bloem, R., Nickovic, D., Roeck, F.: A counting semantics for monitoring LTL specifications over finite traces. In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 547–564. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96145-3_29

    CrossRef  Google Scholar 

  7. Bauer, A., Leucker, M., Schallhart, C.: Monitoring of real-time properties. In: Arun-Kumar, S., Garg, N. (eds.) FSTTCS 2006. LNCS, vol. 4337, pp. 260–272. Springer, Heidelberg (2006). https://doi.org/10.1007/11944836_25

    CrossRef  MATH  Google Scholar 

  8. Bauer, A., Leucker, M., Schallhart, C.: Comparing LTL semantics for runtime verification. J. Logic Comput. 20(3), 651–674 (2010)

    MathSciNet  CrossRef  Google Scholar 

  9. Bauer, A., Leucker, M., Schallhart, C.: Runtime verification for LTL and TLTL. ACM Trans. Softw. Eng. Methodol. (TOSEM) 20(4), 14 (2011)

    CrossRef  Google Scholar 

  10. Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal specifications: a theory and a tool. Softw. Eng. J. 6(6), 387–405 (1991)

    CrossRef  Google Scholar 

  11. Burnstein, I.: Practical Software Testing: A Process-oriented Approach. Springer, Heidelberg (2006)

    MATH  Google Scholar 

  12. Cimatti, A., Clarke, E., Giunchiglia, E., Giunchiglia, F., Pistore, M., Roveri, M., Sebastiani, R., Tacchella, A.: NuSMV 2: an opensource tool for symbolic model checking. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 359–364. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45657-0_29

    CrossRef  Google Scholar 

  13. Duret-Lutz, A., Lewkowicz, A., Fauchille, A., Michaud, T., Renault, É., Xu, L.: Spot 2.0—a framework for LTL and \(\omega \)-automata manipulation. In: Artho, C., Legay, A., Peled, D. (eds.) ATVA 2016. LNCS, vol. 9938, pp. 122–129. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46520-3_8

    CrossRef  Google Scholar 

  14. Eisner, C., Fisman, D., Havlicek, J., Lustig, Y., McIsaac, A., Van Campenhout, D.: Reasoning with temporal logic on truncated paths. In: Hunt, W.A., Somenzi, F. (eds.) CAV 2003. LNCS, vol. 2725, pp. 27–39. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-45069-6_3

    CrossRef  Google Scholar 

  15. Hsueh, M.C., Tsai, T.K., Iyer, R.K.: Fault injection techniques and tools. Computer 30(4), 75–82 (1997)

    CrossRef  Google Scholar 

  16. Jacobs, S., Klein, F., Schirmer, S.: A high-level LTL synthesis format: Tlsf v1. 1. arXiv preprint arXiv:1604.02284 (2016)

  17. Jard, C., Jéron, T.: TGV: theory, principles and algorithms. Int. J. Softw. Tools Technol. Transfer 7(4), 297–315 (2005)

    CrossRef  Google Scholar 

  18. Koch, B., Grabowski, J., Hogrefe, D., Schmitt, M.: Autolink-a tool for automatic test generation from SDL specifications. In: Proceedings. 2nd IEEE Workshop on Industrial Strength Formal Specification Techniques, pp. 114–125. IEEE (1998)

    Google Scholar 

  19. Krichen, M., Tripakis, S.: Black-box conformance testing for real-time systems. In: Graf, S., Mounier, L. (eds.) SPIN 2004. LNCS, vol. 2989, pp. 109–126. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24732-6_8

    CrossRef  Google Scholar 

  20. Li, J., Conradi, R., Bunse, C., Torchiano, M., Slyngstad, O.P.N., Morisio, M.: Development with off-the-shelf components: 10 facts. IEEE Softw. 26(2), 80–87 (2009)

    CrossRef  Google Scholar 

  21. Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems: Safety. Springer, Heidelberg (2012). https://doi.org/10.1007/978-1-4612-4222-2

    CrossRef  MATH  Google Scholar 

  22. Masin, M., Palumbo, F., Myrhaug, H., de Oliveira Filho, J., Pastena, M., Pelcat, M., Raffo, L., Regazzoni, F., Sanchez, A., Toffetti, A., et al.: Cross-layer design of reconfigurable cyber-physical systems. In: Proceedings of the Conference on Design, Automation & Test in Europe, pp. 740–745. European Design and Automation Association (2017)

    Google Scholar 

  23. Luttenberger, M., Meyer, P.J., Sickert, S.: Strix (2018). https://strix.model.in.tum.de/. Accessed 27 June 2019

  24. Palumbo, F., et al.: CERBERO: cross-layer model-based framework for multi-objective design of reconfigurable systems in uncertain hybrid environments: Invited paper: CERBERO teams from UniSS, UniCA, IBM research, TASE, INSA-Rennes, UPM, USI, Abinsula, Ambiesense, TNO, S&T, CRF. In: Proceedings of the 16th ACM International Conference on Computing Frontiers, pp. 320–325. ACM (2019)

    Google Scholar 

  25. Pnueli, A.: The temporal logic of programs. In: 18th Annual Symposium on Foundations of Computer Science, 1977, pp. 46–57. IEEE (1977)

    Google Scholar 

  26. Schmitt, M., Ebner, M., Grabowski, J.: Test generation with autolink and test composer. In: Proceedings of 2nd Workshop of the SDL Forum Society on SDL and MSC-SAM, vol. 2000 (2000)

    Google Scholar 

  27. Tan, L., Sokolsky, O., Lee, I.: Specification-based testing with linear temporal logic. In: Conference on Information Reuse and Integration, pp. 483–498 (2004)

    Google Scholar 

  28. Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., San Francisco (2007)

    Google Scholar 

  29. Zeng, B., Tan, L.: Test reactive systems with Büchi automata: acceptance condition coverage criteria and performance evaluation. In: 2015 IEEE International Conference on Information Reuse and Integration, pp. 380–387. IEEE (2015)

    Google Scholar 

  30. Zeng, B., Tan, L.: Test reactive systems with Büchi-automaton-based temporal requirements. In: Bouabana-Tebibel, T., Rubin, S.H. (eds.) Theoretical Information Reuse and Integration. AISC, vol. 446, pp. 31–57. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-31311-5_2

    CrossRef  Google Scholar 

Download references

Acknowledgments

The research of Luca Pulina and Simone Vuotto is part of the FitOptiVis project funded by the ECSEL Joint Undertaking under grant number H2020-ECSEL-2017-2-783162. The research of Luca Pulina has been also partially funded by the ECSEL JU Project COMP4DRONES and the Sardinian Regional Projects PROSSIMO (POR FESR Sardegna 2014/20-ASSE I).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Massimo Narizzano .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Narizzano, M., Pulina, L., Tacchella, A., Vuotto, S. (2020). Automated Requirements-Based Testing of Black-Box Reactive Systems. In: Lee, R., Jha, S., Mavridou, A., Giannakopoulou, D. (eds) NASA Formal Methods. NFM 2020. Lecture Notes in Computer Science(), vol 12229. Springer, Cham. https://doi.org/10.1007/978-3-030-55754-6_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-55754-6_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-55753-9

  • Online ISBN: 978-3-030-55754-6

  • eBook Packages: Computer ScienceComputer Science (R0)