Skip to main content
SpringerLink
Log in

Learning Monitorable Operational Design Domains for Assured Autonomy

  • Conference paper
  • First Online:
Automated Technology for Verification and Analysis (ATVA 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13505))

Abstract

AI-based autonomous systems are increasingly relying on machine learning (ML) components to perform a variety of complex tasks in perception, prediction, and control. The use of ML components is projected to grow and with it the concern of using these components in systems that operate in safety-critical settings. To guarantee a safe operation of autonomous systems, it is important to run an ML component in its operational design domain (ODD), i.e., the conditions under which using the component does not endanger the safety of the system. Building safe and reliable autonomous systems which may use machine-learning-based components, calls therefore for automated techniques that allow to systematically capture the ODD of systems.

In this paper, we present a framework for learning runtime monitors that capture the ODDs of black-box systems. A runtime monitor of an ODD predicts based on a sequence of monitorable observations whether the system is about to exit the ODD. We particularly investigate the learning of optimal monitors based on counterexample-guided refinement and conformance testing. We evaluate the applicability of our approach on a case study from the domain of autonomous driving.

This work is partially supported by NSF grants 1545126 (VeHICaL), 1646208 and 1837132, by the DARPA contracts FA8750-18-C-0101 (AA) and FA8750-20-C-0156 (SDCPS), by Berkeley Deep Drive, by C3DTI, by the Toyota Research Institute, and by Toyota under the iCyPhy center.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We choose a Boolean codomain for monitors for simplicity reasons. Our approach can be extended easily to quantitative domains, i.e., monitors with a robustness semantics [9, 11].

  2. 2.

    https://scikit-learn.org/.

References

  1. Abdar, M., et al.: A review of uncertainty quantification in deep learning: techniques, applications and challenges. Inf. Fusion 76, 243–297 (2021)

    Article  Google Scholar 

  2. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D.: Concrete problems in AI safety. CoRR, abs/1606.06565 (2016)

    Google Scholar 

  3. Azad, A.S., et al.: Scenic4rl: programmatic modeling and generation of reinforcement learning environments. CoRR, abs/2106.10365 (2021)

    Google Scholar 

  4. Basin, D., Klaedtke, F., Müller, S., Zălinescu, E.: Monitoring metric first-order temporal properties. J. ACM 62(2), 1–45 (2015)

    Article  MathSciNet  Google Scholar 

  5. Blumenthal, M.S., Fraade-Blanar, L., Best, R., Irwin, J.L.: Safe Enough: Approaches to Assessing Acceptable Safety for Automated Vehicles. RAND Corporation, Santa Monica, CA (2020)

    Google Scholar 

  6. Bortolussi, L., Cairoli, F., Paoletti, N., Smolka, S.A., Stoller, S.D.: Neural predictive monitoring and a comparison of frequentist and bayesian approaches. Int. J. Softw. Tools Technol. Transf. 23(4), 615–640 (2021). https://doi.org/10.1007/s10009-021-00623-1

    Article  Google Scholar 

  7. Colwell, I., Phan, B., Saleem, S., Salay, R., Czarnecki, K.: An automated vehicle safety concept based on runtime restriction of the operational design domain. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1910–1917 (2018)

    Google Scholar 

  8. Desai, A., Ghosh, S., Seshia, S.A., Shankar, N., Tiwari, A.: SOTER: a runtime assurance framework for programming safe robotics systems. In: IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) (2019)

    Google Scholar 

  9. Deshmukh, J.V., Donzé, A., Ghosh, S., Jin, X., Juniwal, G., Seshia, S.A.: Robust online monitoring of signal temporal logic. Formal Meth. Syst. Des. 51(1), 5–30 (2017). https://doi.org/10.1007/s10703-017-0286-7

    Article  MATH  Google Scholar 

  10. Dietterich, T.G., Horvitz, E.: Rise of concerns about AI: reflections and directions. Commun. ACM 58(10), 38–40 (2015)

    Article  Google Scholar 

  11. Donzé, A., Maler, O.: Robust satisfaction of temporal logic over real-valued signals. In: Chatterjee, K., Henzinger, T.A. (eds.) FORMATS 2010. LNCS, vol. 6246, pp. 92–106. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15297-9_9

    Chapter  MATH  Google Scholar 

  12. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning, pp. 1–16 (2017)

    Google Scholar 

  13. Dreossi, T., et al.: VerifAI: a toolkit for the formal design and analysis of artificial intelligence-based systems. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 432–442. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_25

    Chapter  Google Scholar 

  14. Dreossi, T., Jha, S., Seshia, S.A.: Semantic adversarial deep learning. In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 3–26. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96145-3_1

    Chapter  Google Scholar 

  15. Falcone, Y., Mounier, L., Fernandez, J.-C., Richier, J.-L.: Runtime enforcement monitors: composition, synthesis, and enforcement abilities. Formal Meth. Syst. Des. 38(3), 223–262 (2011). https://doi.org/10.1007/s10703-011-0114-4

    Article  MATH  Google Scholar 

  16. Faymonville, P., et al.: StreamLAB: stream-based monitoring of cyber-physical systems. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 421–431. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_24

    Chapter  Google Scholar 

  17. Finkbeiner, B., Sipma, H.: Checking finite traces using alternating automata. Formal Meth. Syst. Des. 24(2), 101–127 (2004). https://doi.org/10.1023/B:FORM.0000017718.28096.48

    Article  MATH  Google Scholar 

  18. Fremont, D.J., Chiu, J., Margineantu, D.D., Osipychev, D., Seshia, S.A.: Formal analysis and redesign of a neural network-based aircraft taxiing system with VerifAI. In: Lahiri, S.K., Wang, C. (eds.) CAV 2020. LNCS, vol. 12224, pp. 122–134. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-53288-8_6

    Chapter  Google Scholar 

  19. Fremont, D.J., et al.: Scenic: a language for scenario specification and scene generation. In: PLDI, pp. 63–78. ACM (2019)

    Google Scholar 

  20. Fremont, D.J., et al.: Scenic: a language for scenario specification and data generation (2020)

    Google Scholar 

  21. Fremont, D.J., et al.: Formal scenario-based testing of autonomous vehicles: from simulation to the real world. In: ITSC (2020)

    Google Scholar 

  22. Gawlikowski, J., et al.: A survey of uncertainty in deep neural networks. CoRR, abs/2107.03342 (2021)

    Google Scholar 

  23. Ghosh, S., Pant, Y.V., Ravanbakhsh, H., Seshia, S.A.: Counterexample-guided synthesis of perception models and control. In: American Control Conference (ACC), pp. 3447–3454. IEEE (2021)

    Google Scholar 

  24. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)

    Article  MathSciNet  Google Scholar 

  25. The British Standards Institution. Operational design domain (odd) taxonomy for an automated driving system (ads) - specification. BSI PAS 1883 (2020)

    Google Scholar 

  26. Irvine, P., Zhang, X., Khastgir, S., Schwalb, E., Jennings, P.: A two-level abstraction ODD definition language: part i*. In: 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2614–2621. IEEE Press (2021)

    Google Scholar 

  27. Jha, S., Gulwani, S., Seshia, S.A., Tiwari, A.: Oracle-guided component-based program synthesis. In: ICSE, vol. 1, pp. 215–224. ACM (2010)

    Google Scholar 

  28. Jha, S., Seshia, S.A.: A theory of formal synthesis via inductive learning. Acta Informatica 54(7), 693–726 (2017). https://doi.org/10.1007/s00236-017-0294-5

    Article  MathSciNet  MATH  Google Scholar 

  29. Khastgir, S., Birrell, S.A., Dhadyalla, G., Jennings, P.A.: Calibrating trust through knowledge: introducing the concept of informed safety for automation in vehicles. In: Transportation Research Part C: Emerging Technologies (2018)

    Google Scholar 

  30. Khastgir, S., Brewerton, S., Thomas, J., Jennings, P.: Systems approach to creating test scenarios for automated driving systems. Reliab. Eng. Syst. Saf. 215, 107610 (2021)

    Google Scholar 

  31. Koymans, R.: Specifying real-time properties with metric temporal logic. Real-Time Syst. 2(4), 255–299 (1990)

    Article  Google Scholar 

  32. Lee, I., Kannan, S., Kim, M., Sokolsky, O., Viswanathan, M.: Runtime assurance based on formal specifications. In: Arabnia, H.R. (ed.) Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications, PDPTA 1999, June 28 - Junlly 1 1999, Las Vegas, Nevada, USA, pp. 279–287. CSREA Press (1999)

    Google Scholar 

  33. Lukina, A., Schilling, C., Henzinger, T.A.: Into the unknown: active monitoring of neural networks. In: Feng, L., Fisman, D. (eds.) RV 2021. LNCS, vol. 12974, pp. 42–61. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88494-9_3

    Chapter  Google Scholar 

  34. Maler, O., Nickovic, D.: Monitoring temporal properties of continuous signals. In: Lakhnech, Y., Yovine, S. (eds.) FORMATS/FTRTFT -2004. LNCS, vol. 3253, pp. 152–166. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30206-3_12

    Chapter  MATH  Google Scholar 

  35. Mitsch, S., Platzer, A.: Modelplex: verified runtime validation of verified cyber-physical system models. Formal Meth. Syst. Des. 49(1–2), 33–74 (2016). https://doi.org/10.1007/s10703-016-0241-z

    Article  MATH  Google Scholar 

  36. SAE on Road Automated Driving Committee et al. SAE J3016. taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Technical report

    Google Scholar 

  37. Roşu, G., Chen, F., Ball, T.: Synthesizing monitors for safety properties: this time with calls and returns. In: Leucker, M. (ed.) RV 2008. LNCS, vol. 5289, pp. 51–68. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89247-2_4

    Chapter  Google Scholar 

  38. Sánchez, C., et al.: A survey of challenges for runtime verification from advanced application domains (beyond software). Formal Meth. Syst. Des. 54(3), 279–335 (2019). https://doi.org/10.1007/s10703-019-00337-w

  39. Seshia, S.A.: Introspective environment modeling. In: Finkbeiner, B., Mariani, L. (eds.) RV 2019. LNCS, vol. 11757, pp. 15–26. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32079-9_2

    Chapter  Google Scholar 

  40. Seshia, S.A., Sadigh, D.: Towards verified artificial intelligence. CoRR, abs/1606.08514 (2016)

    Google Scholar 

  41. Sha, L.: Using simplicity to control complexity. IEEE Softw. 18(4), 20–28 (2001)

    Article  Google Scholar 

  42. Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, USA (2014)

    Book  Google Scholar 

  43. Shivakumar, S., Torfah, H., Desai, A., Seshia, S.A.: SOTER on ROS: a run-time assurance framework on the robot operating system. In: Deshmukh, J., Ničković, D. (eds.) RV 2020. LNCS, vol. 12399, pp. 184–194. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60508-7_10

    Chapter  Google Scholar 

  44. Thorn, E., Kimmel, S. C., Chaka, M..: A framework for automated driving system testable cases and scenarios (2018)

    Google Scholar 

  45. Torfah, H.: Stream-based monitors for real-time properties. In: Finkbeiner, B., Mariani, L. (eds.) RV 2019. LNCS, vol. 11757, pp. 91–110. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32079-9_6

    Chapter  Google Scholar 

  46. Torfah, H., Junges, S., Fremont, D.J., Seshia, S.A.: Formal analysis of AI-based autonomy: from modeling to runtime assurance. In: Feng, L., Fisman, D. (eds.) RV 2021. LNCS, vol. 12974, pp. 311–330. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88494-9_19

    Chapter  MATH  Google Scholar 

  47. Zhang, X., Khastgir, S., Jennings, P.: Scenario description language for automated driving systems: a two level abstraction approach. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 973–980 (2020)

    Google Scholar 

Download references

Acknowledgments

The authors are grateful to Daniel Fremont for his contributions to the VerifAI and Scenic projects, and assistance with these tools for this paper. The authors also want to thank Johnathan Chiu, Tommaso Dreossi, Shromona Ghosh, Francis Indaheng, Edward Kim, Hadi Ravanbakhsh, Ameesh Shah and Kesav Viswanadha for their valuable feedback and contributions to the VerifAI project.

Author information

Authors and Affiliations

  1. University of California at Berkeley, Berkeley, USA

    Hazem Torfah, Carol Xie, Marcell Vazquez-Chanlatte & Sanjit A. Seshia

  2. Radboud University, Nijmegen, The Netherlands

    Sebastian Junges

Authors
  1. Hazem Torfah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carol Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sebastian Junges
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marcell Vazquez-Chanlatte
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sanjit A. Seshia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hazem Torfah .

Editor information

Editors and Affiliations

  1. Université Paris Diderot, Paris, France

    Ahmed Bouajjani

  2. Brno University of Technology, Brno, Czech Republic

    Lukáš Holík

  3. Chinese Academy of Sciences, Beijing, China

    Zhilin Wu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Torfah, H., Xie, C., Junges, S., Vazquez-Chanlatte, M., Seshia, S.A. (2022). Learning Monitorable Operational Design Domains for Assured Autonomy. In: Bouajjani, A., Holík, L., Wu, Z. (eds) Automated Technology for Verification and Analysis. ATVA 2022. Lecture Notes in Computer Science, vol 13505. Springer, Cham. https://doi.org/10.1007/978-3-031-19992-9_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19992-9_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19991-2

  • Online ISBN: 978-3-031-19992-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation