Skip to main content

Automated Translation of Natural Language Requirements to Runtime Monitors

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13243)

Abstract

Runtime verification (RV) enables monitoring systems at runtime, to detect property violations early and limit their potential consequences. This paper presents an end-to-end framework to capture requirements in structured natural language and generate monitors that capture their semantics faithfully. We leverage NASA’s Formal Requirement Elicitation Tool (fret), and the RV system Copilot. We extend fret with mechanisms to capture additional information needed to generate monitors, and introduce Ogma, a new tool to bridge the gap between fret and Copilot. With this framework, users can write requirements in an intuitive format and obtain real-time C monitors suitable for use in embedded systems. Our toolchain is available as open source.

Author contributed to this work prior to joining AWS.

References

  1. Copilot. https://github.com/Copilot-Language/copilot/. Accessed Oct 04, 2021

  2. FRET: Formal Requirements Elicitation Tool. https://github.com/NASA-SW-VnV/fret/. Accessed Oct 04, 2021

  3. FRET: Formal Requirements Elicitation Tool - User Manual. https://github.com/NASA-SW-VnV/fret/blob/master/fret-electron/docs/_media/userManual.md. See Section “Exporting for Analysis”. Accessed Oct 04, 2021

  4. Ogma. https://github.com/nasa/ogma/. Accessed Oct 04, 2021

  5. E. Bartocci, Y. Falcone, A. Francalanza, and G. Reger. Introduction to runtime verification. In Lectures on Runtime Verification - Introductory and Advanced Topics, volume 10457 of Lecture Notes in Computer Science, pages 1–33. Springer, 2018

    Google Scholar 

  6. J. Baumeister, B. Finkbeiner, S. Schirmer, M. Schwenger, and C. Torens. RTLola cleared for take-off: Monitoring autonomous aircraft. In S. K. Lahiri and C. Wang, editors, Computer Aided Verification, pages 28–39, Cham, 2020. Springer International Publishing

    Google Scholar 

  7. S. Biewer, B. Finkbeiner, H. Hermanns, M. A. Köhl, Y. Schnitzer, and M. Schwenger. RTLola on board: Testing real driving emissions on your phone. In J. F. Groote and K. G. Larsen, editors, Tools and Algorithms for the Construction and Analysis of Systems, pages 365–372, Cham, 2021. Springer International Publishing

    Google Scholar 

  8. A. Boteanu, T. Howard, J. Arkin, and H. Kress-Gazit. A model for verifiable grounding and execution of complex natural language instructions. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2649–2654, 2016

    Google Scholar 

  9. K. Claessen and J. Hughes. QuickCheck: a lightweight tool for random testing of Haskell programs. ACM Sigplan Notices, 46(4):53–64, 2011

    Google Scholar 

  10. D. Cohen, M. S. Feather, K. Narayanaswamy, and S. S. Fickas. Automatic monitoring of software requirements. In Proceedings of the 19th International Conference on Software Engineering, pages 602–603, 1997

    Google Scholar 

  11. C. Elliott. On example models and challenges ahead for the evaluation of complex cyber-physical systems with state of the art formal methods V&V, Lockheed Martin Skunk Works. In A. F. R. Laboratory, editor, Safe & Secure Systems and Software Symposium (S5), 2015

    Google Scholar 

  12. C. Elliott. An example set of cyber-physical V&V challenges for S5, Lockheed Martin Skunk Works. In A. F. R. Laboratory, editor, Safe & Secure Systems and Software Symposium (S5), 2016

    Google Scholar 

  13. P. Faymonville, B. Finkbeiner, M. Schledjewski, M. Schwenger, M. Stenger, L. Tentrup, and H. Torfah. StreamLAB: Stream-based monitoring of cyber-physical systems. In I. Dillig and S. Tasiran, editors, Computer Aided Verification, pages 421–431, Cham, 2019. Springer International Publishing

    Google Scholar 

  14. A. Fifarek, L. G. Wagner, J. A. Hoffman, B. D. Rodes, M. A. Aiello, and J. A. Davis. SpeAR v2.0: Formalized past LTL specification and analysis of requirements. In NASA Formal Methods - 9th International Symposium, NFM 2017, Moffett Field, CA, USA, May 16–18, 2017, Proceedings, pages 420–426, 2017

    Google Scholar 

  15. A. Gacek, J. Backes, M. Whalen, L. Wagner, and E. Ghassabani. The jk ind model checker. In International Conference on Computer Aided Verification, pages 20–27. Springer, 2018

    Google Scholar 

  16. S. Ghosh, D. Elenius, W. Li, P. Lincoln, N. Shankar, and W. Steiner. ARSENAL: automatic requirements specification extraction from natural language. In S. Rayadurgam and O. Tkachuk, editors, NASA Formal Methods - 8th International Symposium, NFM 2016, Minneapolis, MN, USA, June 7-9, 2016, Proceedings, volume 9690 of Lecture Notes in Computer Science, pages 41–46. Springer, 2016

    Google Scholar 

  17. D. Giannakopoulou, T. Pressburger, A. Mavridou, J. Rhein, J. Schumann, and N. Shi. Formal requirements elicitation with FRET. In Joint Proceedings of REFSQ-2020 Workshops, Doctoral Symposium, Live Studies Track, and Poster Track co-located with the 26th International Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2020), 2020

    Google Scholar 

  18. D. Giannakopoulou, T. Pressburger, A. Mavridou, and J. Schumann. Automated formalization of structured natural language requirements. Inf. Softw. Technol., 137:106590, 2021

    Google Scholar 

  19. A. Goodloe and L. Pike. Monitoring distributed real-time systems: A survey and future directions. Technical Report NASA/CR-2010-216724, NASA Langley Research Center, July 2010

    Google Scholar 

  20. N. Halbwachs, P. Caspi, P. Raymond, and D. Pilaud. The synchronous data flow programming language LUSTRE. Proceedings of the IEEE, 79(9):1305–1320, 1991

    Google Scholar 

  21. K. Havelund and A. Goldberg. Verify Your Runs, pages 374–383. Springer, Berlin Heidelberg, Berlin, Heidelberg, 2008

    Google Scholar 

  22. B. Jeannet and F. Gaucher. Debugging embedded systems requirements with STIMULUS: an automotive case-study. In 8th European Congress on Embedded Real Time Software and Systems (ERTS 2016), Toulouse, France, Jan. 2016

    Google Scholar 

  23. J. C. Knight. Safety critical systems: Challenges and directions. In Proceedings of the 24th International Conference on Software Engineering, ICSE ’02, pages 547–550. ACM, 2002

    Google Scholar 

  24. C. Lignos, V. Raman, C. Finucane, M. Marcus, and H. Kress-Gazit. Provably correct reactive control from natural language. Auton. Robots, 38(1):89–105, jan 2015

    Google Scholar 

  25. L. Lúcio, S. Rahman, C.-H. Cheng, and A. Mavin. Just formal enough? Automated analysis of EARS requirements. In NASA Formal Methods - 9th International Symposium, NFM 2017, Moffett Field, CA, USA, May 16-18, 2017, Proceedings, pages 427–434, May 2017

    Google Scholar 

  26. A. Mavridou, H. Bourbouh, P. L. Garoche, and M. Hejase. Evaluation of the FRET and CoCoSim tools on the ten Lockheed Martin cyber-physical challenge problems. Technical Report TM-2019-220374, National Aeronautics and Space Administration, February 2020

    Google Scholar 

  27. A. Mavridou, H. Bourbouh, D. Giannakopoulou, T. Pressburger, M. Hejase, P.-L. Garoche, and J. Schumann. The ten Lockheed Martin cyber-physical challenges: Formalized, analyzed, and explained. In 2020 IEEE 28th International Requirements Engineering Conference (RE), pages 300–310, 2020

    Google Scholar 

  28. P. Moosbrugger, K. Y. Rozier, and J. Schumann. R2U2: monitoring and diagnosis of security threats for unmanned aerial systems. Formal Methods in System Design, 51(1):31–61, 2017

    Google Scholar 

  29. I. Perez, F. Dedden, and A. Goodloe. Copilot 3. Technical Report NASA/TM–2020–220587, NASA Langley Research Center, April 2020

    Google Scholar 

  30. I. Perez, A. Goodloe, and W. Edmonson. Fault-tolerant swarms. In 2019 IEEE International Conference on Space Mission Challenges for Information Technology (SMC-IT), pages 47–54. IEEE, 2019

    Google Scholar 

  31. I. Perez and A. E. Goodloe. Fault-tolerant functional reactive programming (extended version). Journal of Functional Programming, 30, 2020

    Google Scholar 

  32. I. Perez, A. Mavridou, T. Pressburger, A. Goodloe, and D. Giannakopoulou. Artifact for Automated Translation of Natural Language Requirements to Runtime Monitors. https://doi.org/10.5281/zenodo.5888956. Accessed Jan 21, 2022

  33. I. Perez, A. Mavridou, T. Pressburger, A. Goodloe, and D. Giannakopoulou. Integrating FRET with Copilot: Automated Translation of Natural Language Requirements to Runtime Monitors. Technical Report NASA/TM–20220000049, NASA, January 2022

    Google Scholar 

  34. I. Perez and H. Nilsson. Runtime verification and validation of functional reactive systems. Journal of Functional Programming, 30:e28, 2020

    Google Scholar 

  35. L. Pike, A. Goodloe, R. Morisset, and S. Niller. Copilot: A hard real-time runtime monitor. In Proceedings of the 1st Intl. Conference on Runtime Verification, LNCS. Springer, November 2010

    Google Scholar 

  36. L. Pike, N. Wegmann, S. Niller, and A. Goodloe. Copilot: Monitoring embedded systems. Innov. Syst. Softw. Eng., 9(4):235–255, Dec. 2013

    Google Scholar 

  37. T. Reinbacher, K. Y. Rozier, and J. Schumann. Temporal-logic based runtime observer pairs for system health management of real-time systems. In E. Ábrahám and K. Havelund, editors, Tools and Algorithms for the Construction and Analysis of Systems, pages 357–372, Berlin, Heidelberg, 2014. Springer, Berlin Heidelberg

    Google Scholar 

  38. J. Schumann, P. Moosbrugger, and K. Y. Rozier. R2u2: monitoring and diagnosis of security threats for unmanned aerial systems. In Runtime Verification, pages 233–249. Springer, 2015

    Google Scholar 

  39. H. Torfah. Stream-based monitors for real-time properties. In B. Finkbeiner and L. Mariani, editors, Runtime Verification - 19th International Conference, RV 2019, Porto, Portugal, October 8-11, 2019, Proceedings, volume 11757 of Lecture Notes in Computer Science, pages 91–110. Springer, 2019

    Google Scholar 

  40. J. Wilmot. A core flight software system. In Proceedings of the 3rd IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis, CODES+ISSS ’05, pages 13–14, New York, NY, USA, 2005. ACM

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Ivan Perez or Anastasia Mavridou .

Editor information

Editors and Affiliations

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and Permissions

Copyright information

© 2022 The Author(s)

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Perez, I., Mavridou, A., Pressburger, T., Goodloe, A., Giannakopoulou, D. (2022). Automated Translation of Natural Language Requirements to Runtime Monitors. In: Fisman, D., Rosu, G. (eds) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2022. Lecture Notes in Computer Science, vol 13243. Springer, Cham. https://doi.org/10.1007/978-3-030-99524-9_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-99524-9_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-99523-2

  • Online ISBN: 978-3-030-99524-9

  • eBook Packages: Computer ScienceComputer Science (R0)