Skip to main content
SpringerLink
Log in

Formal Analysis of AI-Based Autonomy: From Modeling to Runtime Assurance

  • Conference paper
  • First Online:
Book cover Runtime Verification (RV 2021)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 12974))

Included in the following conference series:

Abstract

Autonomous systems are increasingly deployed in safety-critical applications and rely more on high-performance components based on artificial intelligence (AI) and machine learning (ML). Runtime monitors play an important role in raising the level of assurance in AI/ML-based autonomous systems by ensuring that the autonomous system stays safe within its operating environment. In this tutorial, we present VerifAI, an open-source toolkit for the formal design and analysis of systems that include AI/ML components. VerifAI provides features supporting a variety of use cases including formal modeling of the autonomous system and its environment, automatic falsification of system-level specifications as well as other simulation-based verification and testing methods, automated diagnosis of errors, and automatic specification-driven parameter and component synthesis. In particular, we describe the use of VerifAI for generating runtime monitors that capture the safe operational environment of systems with AI/ML components. We illustrate the advantages and applicability of VerifAI in real-life applications using a case study from the domain of autonomous aviation.

This work is partially supported by NSF grants 1545126 (VeHICaL), 1646208 and 1837132, by the DARPA contracts FA8750-18-C-0101 (AA) and FA8750-20-C-0156 (SDCPS), by Berkeley Deep Drive, by the Toyota Research Institute, and by Toyota under the iCyPhy center.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aarts, F., Jonsson, B., Uijen, J., Vaandrager, F.: Generating models of infinite-state communication protocols using regular inference with abstraction. Formal Methods Syst. Des. 46(1), 1–41 (2014). https://doi.org/10.1007/s10703-014-0216-x

    Article  MATH  Google Scholar 

  2. Aceto, L., Cassar, I., Francalanza, A., Ingólfsdóttir, A.: On runtime enforcement via suppressions. In: CONCUR. LIPIcs, vol. 118, pp. 34:1–34:17 (2018)

    Google Scholar 

  3. Azad, A.S., et al.: Scenic4RL: programmatic modeling and generation of reinforcement learning environments. CoRR, abs/2106.10365 (2021)

    Google Scholar 

  4. Baumeister, J., Finkbeiner, B., Schwenger, M., Torfah, H.: FPGA stream-monitoring of real-time properties. ACM Trans. Embed. Comput. Syst. 18(5s), 88:1–88:24 (2019)

    Google Scholar 

  5. Bortolussi, L., Cairoli, F., Paoletti, N., Smolka, S.A., Stoller, S.D.: Neural predictive monitoring. In: Finkbeiner, B., Mariani, L. (eds.) RV 2019. LNCS, vol. 11757, pp. 129–147. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32079-9_8

    Chapter  Google Scholar 

  6. Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Wadsworth (1984)

    Google Scholar 

  7. Cairoli, F., Bortolussi, L., Paoletti, N.: Neural predictive monitoring under partial observability. CoRR, abs/2108.07134 (2021)

    Google Scholar 

  8. Chou, Y., Yoon, H., Sankaranarayanan, S.: Predictive runtime monitoring of vehicle models using Bayesian estimation and reachability analysis. In: IROS, pp. 2111–2118. IEEE (2020)

    Google Scholar 

  9. Desai, A., Ghosh, S., Seshia, S.A., Shankar, N., Tiwari, A.: SOTER: a runtime assurance framework for programming safe robotics systems. In: IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) (2019)

    Google Scholar 

  10. Deshmukh, J.V., Donzé, A., Ghosh, S., Jin, X., Juniwal, G., Seshia, S.A.: Robust online monitoring of signal temporal logic. Formal Methods Syst. Des. 51(1), 5–30 (2017). https://doi.org/10.1007/s10703-017-0286-7

    Article  MATH  Google Scholar 

  11. Dreossi, T., et al.: VerifAI: a toolkit for the formal design and analysis of artificial intelligence-based systems. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 432–442. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_25

    Chapter  Google Scholar 

  12. Dreossi, T., Jha, S., Seshia, S.A.: Semantic adversarial deep learning. In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 3–26. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96145-3_1

    Chapter  Google Scholar 

  13. Falcone, Y., Mounier, L., Fernandez, J.-C., Richier, J.-L.: Runtime enforcement monitors: composition, synthesis, and enforcement abilities. Formal Methods Syst. Des. 38(3), 223–262 (2011)

    Article  Google Scholar 

  14. Faymonville, P., et al.: StreamLAB: stream-based monitoring of cyber-physical systems. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 421–431. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_24

    Chapter  Google Scholar 

  15. Finkbeiner, B., Sipma, H.: Checking finite traces using alternating automata. Form. Methods Syst. Des. 24(2), 101–127 (2004)

    Article  Google Scholar 

  16. Fremont, D.J., Chiu, J., Margineantu, D.D., Osipychev, D., Seshia, S.A.: Formal analysis and redesign of a neural network-based aircraft taxiing system with VerifAI. In: Lahiri, S.K., Wang, C. (eds.) CAV 2020. LNCS, vol. 12224, pp. 122–134. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-53288-8_6

    Chapter  Google Scholar 

  17. Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: PLDI (2019)

    Google Scholar 

  18. Fremont, D.J., et al.: Scenic: a language for scenario specification and data generation (2020)

    Google Scholar 

  19. Fremont, D.J., et al.: Formal scenario-based testing of autonomous vehicles: from simulation to the real world. In: ITSC (2020)

    Google Scholar 

  20. Ghosh, S., Pant, Y.V., Ravanbakhsh, H., Seshia, S.A.: Counterexample-guided synthesis of perception models and control. In: American Control Conference (ACC), pp. 3447–3454. IEEE (2021)

    Google Scholar 

  21. Havelund, K., Roşu, G.: Synthesizing monitors for safety properties. In: Katoen, J.-P., Stevens, P. (eds.) TACAS 2002. LNCS, vol. 2280, pp. 342–356. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-46002-0_24

    Chapter  MATH  Google Scholar 

  22. Henzinger, T.A., Ho, P.-H., Wong-Toi, H.: Algorithmic analysis of nonlinear hybrid systems. IEEE Trans. Autom. Control 43(4), 540–554 (1998)

    Article  MathSciNet  Google Scholar 

  23. Isberner, M., Steffen, B., Howar, F.: LearnLib tutorial - an open-source Java library for active automata learning. In: Bartocci, E., Majumdar, R. (eds.) RV 2015. LNCS, vol. 9333, pp. 358–377. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23820-3_25

    Chapter  Google Scholar 

  24. Jha, S., Gulwani, S., Seshia, S.A., Tiwari, A.: Oracle-guided component-based program synthesis. In: ICSE (1), pp. 215–224. ACM (2010)

    Google Scholar 

  25. Jha, S., Seshia, S.A.: A theory of formal synthesis via inductive learning. Acta Informatica 54(7), 693–726 (2017). https://doi.org/10.1007/s00236-017-0294-5

    Article  MathSciNet  MATH  Google Scholar 

  26. Junges, S., Torfah, H., Seshia, S.A.: Runtime Monitors for Markov Decision Processes. In: Silva, A., Leino, K.R.M. (eds.) CAV 2021. LNCS, vol. 12760, pp. 553–576. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-81688-9_26

    Chapter  Google Scholar 

  27. Koymans, R.: Specifying real-time properties with metric temporal logic. Real-Time Syst. 2(4), 255–299 (1990)

    Article  Google Scholar 

  28. Leucker, M., Sánchez, C., Scheffel, T., Schmitz, M., Schramm, A.: Tessla: runtime verification of non-synchronized real-time streams. In: SAC, pp. 1925–1933. ACM (2018)

    Google Scholar 

  29. Maler, O., Nickovic, D.: Monitoring temporal properties of continuous signals. In: Lakhnech, Y., Yovine, S. (eds.) FORMATS/FTRTFT -2004. LNCS, vol. 3253, pp. 152–166. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30206-3_12

    Chapter  MATH  Google Scholar 

  30. Mens, I.-E., Maler, O.: Learning regular languages over large ordered alphabets. Log. Methods Comput. Sci. 11(3) (2015)

    Google Scholar 

  31. Phan, D., Yang, J., Grosu, R., Smolka, S.A., Stoller, S.D.: Collision avoidance for mobile robots with limited sensing and limited information about moving obstacles. Formal Methods Syst. Des. 51(1), 62–86 (2017). https://doi.org/10.1007/s10703-016-0265-4

    Article  MATH  Google Scholar 

  32. Pike, L., Goodloe, A., Morisset, R., Niller, S.: Copilot: a hard real-time runtime monitor. In: Barringer, H., et al. (eds.) RV 2010. LNCS, vol. 6418, pp. 345–359. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16612-9_26

    Chapter  Google Scholar 

  33. Pitt, L., Warmuth, M.K.: The minimum consistent DFA problem cannot be approximated within any polynomial. J. ACM 40(1), 95–142 (1993)

    Article  MathSciNet  Google Scholar 

  34. Ross Quinlan, J.: Induction of decision trees. Mach. Learn. 1(1), 81–106 (1986)

    Google Scholar 

  35. Laminar Research. X-Plane 11 (2019). https://www.x-plane.com/

  36. Rivest, R.L.: Learning decision lists. Mach. Learn. 2(3), 229–246 (1987)

    Google Scholar 

  37. Sánchez, C., et al.: A survey of challenges for runtime verification from advanced application domains (beyond software). Formal Methods Syst. Des. 54(3), 279–335 (2019)

    Article  Google Scholar 

  38. Schneider, F.B.: Enforceable security policies. ACM Trans. Inf. Syst. Secur. 3(1), 30–50 (2000)

    Article  Google Scholar 

  39. Seshia, S.A.: Introspective environment modeling. In: Finkbeiner, B., Mariani, L. (eds.) RV 2019. LNCS, vol. 11757, pp. 15–26. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32079-9_2

    Chapter  Google Scholar 

  40. Seshia, S.A., Sadigh, D., Shankar Sastry, S.: Towards Verified Artificial Intelligence. arXiv e-prints (2016)

    Google Scholar 

  41. Seto, D., Ferriera, E., Marz, T.: Case study: development of a baseline controller for automatic landing of an F-16 aircraft using linear matrix inequalities (LMIs). Technical report CMU/SEI-99-TR-020, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA (2000)

    Google Scholar 

  42. Sha, L.: Using simplicity to control complexity. IEEE Softw. 18(4), 20–28 (2001)

    Article  Google Scholar 

  43. Shivakumar, S., Torfah, H., Desai, A., Seshia, S.A.: SOTER on ROS: a run-time assurance framework on the robot operating system. In: Deshmukh, J., Ničković, D. (eds.) RV 2020. LNCS, vol. 12399, pp. 184–194. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60508-7_10

    Chapter  Google Scholar 

  44. Stoller, S.D., et al.: Runtime verification with state estimation. In: Khurshid, S., Sen, K. (eds.) RV 2011. LNCS, vol. 7186, pp. 193–207. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29860-8_15

    Chapter  Google Scholar 

  45. Teubert, C., Watkins, J.: The X-Plane Connect Toolbox (2019). https://github.com/nasa/ XPlaneConnect

  46. Torfah, H.: Stream-based monitors for real-time properties. In: Finkbeiner, B., Mariani, L. (eds.) RV 2019. LNCS, vol. 11757, pp. 91–110. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32079-9_6

    Chapter  Google Scholar 

  47. Torfah, H., Shah, S., Chakraborty, S., Akshay, S., Seshia, S.A.: Synthesizing pareto-optimal interpretations for black-box models. In: FMCAD. IEEE (2021)

    Google Scholar 

  48. Vaandrager, F.W.: Model learning. Commun. ACM 60(2), 86–95 (2017)

    Article  Google Scholar 

  49. Valiant, L.G.: A theory of the learnable. Commun. ACM 27(11), 1134–1142 (1984)

    Article  Google Scholar 

Download references

Acknowledgments

The authors are grateful to Johnathan Chiu, Tommaso Dreossi, Shromona Ghosh, Francis Indaheng, Edward Kim, Hadi Ravanbakhsh, Marcell Vazquez-Chanlatte, and Kesav Viswanadha for their valuable contributions to the VerifAI project. We also thank the team at Boeing helping to define the TaxiNet challenge problem including especially Dragos D. Margineantu and Denis Osipychev.

Author information

Authors and Affiliations

  1. University of California, Berkeley, USA

    Hazem Torfah, Sebastian Junges & Sanjit A. Seshia

  2. University of California, Santa Cruz, USA

    Daniel J. Fremont

Authors
  1. Hazem Torfah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Junges
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel J. Fremont
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sanjit A. Seshia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hazem Torfah .

Editor information

Editors and Affiliations

  1. University of Virginia, Charlottesville, VA, USA

    Lu Feng

  2. Ben-Gurion University of the Negev, Be'er Sheva, Israel

    Dana Fisman

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Torfah, H., Junges, S., Fremont, D.J., Seshia, S.A. (2021). Formal Analysis of AI-Based Autonomy: From Modeling to Runtime Assurance. In: Feng, L., Fisman, D. (eds) Runtime Verification. RV 2021. Lecture Notes in Computer Science(), vol 12974. Springer, Cham. https://doi.org/10.1007/978-3-030-88494-9_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-88494-9_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88493-2

  • Online ISBN: 978-3-030-88494-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation