Abstract
Runtime assurance (RTA) addresses the problem of keeping an autonomous system safe while using an untrusted (or experimental) controller. This can be done via logic that explicitly switches between the untrusted controller and a safety controller, or logic that filters the input provided by the untrusted controller. While several tools implement specific instances of RTAs, there is currently no framework for evaluating different approaches. Given the importance of the RTA problem in building safe autonomous systems, an evalutation tool is needed. In this paper, we present the \(\textsf {RTAEval}\) framework as a low code framework that can be used to quickly evaluate different RTA logics for different types of agents in a variety of scenarios. \(\textsf {RTAEval}\) is designed to quickly create scenarios, run different RTA logics, and collect data that can be used to evaluate and visualize performance. In this paper, we describe different components of \(\textsf {RTAEval}\) and show how it can be used to create and evaluate scenarios involving multiple aircraft models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aiello, A., Berryman, J., Grohs, J., Schierman, J.: Run-time assurance for advanced flight-critical control systems. In: Proceedings of AIAA Guidance, Navigation, and Control Conference, AIAA 2010–8041, Toronto, Ontario Canada, Aug., 2010 (2010)
Ames, A.D., Coogan, S., Egerstedt, M., Notomista, G., Sreenath, K., Tabuada, P.: Control barrier functions: theory and applications. In: 2019 18th European control conference (ECC), pp. 3420–3431. IEEE (2019)
Bak, S., Manamcheri, K., Mitra, S., Caccamo, M.: Sandboxing controllers for cyber-physical systems. In: 2011 IEEE/ACM Second International Conference on Cyber-Physical Systems, pp. 3–12. IEEE (2011)
Cofer, D., et al.: Flight test of a collision avoidance neural network with run-time assurance. In: Digital Avionics Systems Conference (2022)
Desai, A., Ghosh, S., Seshia, S.A., Shankar, N., Tiwari, A.: Soter: a runtime assurance framework for programming safe robotics systems. In: 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 138–150. IEEE (2019)
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: an open urban driving simulator. In: Conference on robot learning, pp. 1–16. PMLR (2017)
Dunlap, K.: Run Time Assurance for Intelligent Aerospace Control Systems. Ph.D. thesis, University of Cincinnati (2022)
Dunlap, K., Hibbard, M., Mote, M., Hobbs, K.: Comparing run time assurance approaches for safe spacecraft docking. IEEE Control Syst. Lett. 6, 1849–1854 (2021)
Dunlap, K., Mote, M., Delsing, K., Hobbs, K.L.: Run time assured reinforcement learning for safe satellite docking. J. Aerosp. Inf. Syst. 20(1), 25–36 (2023)
Fan, C., Miller, K., Mitra, S.: Fast and guaranteed safe controller synthesis for nonlinear vehicle models. In: Lahiri, S.K., Wang, C. (eds.) CAV 2020. LNCS, vol. 12224, pp. 629–652. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-53288-8_31
Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 63–78 (2019)
Hibbard, M., Topcu, U., Hobbs, K.: Guaranteeing safety via active-set invariance filters for multi-agent space systems with coupled dynamics. In: 2022 American Control Conference (ACC), pp. 430–436. IEEE (2022)
Mehmood, U., Sheikhi, S., Bak, S., Smolka, S.A., Stoller, S.D.: The black-box simplex architecture for runtime assurance of autonomous cps. In: Deshmukh, J.V., Havelund, K., Perez, I. (eds.) NFM 2022. LNCS, vol. 13260, pp. 231–250. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-06773-0_12
Mote, M.L., Hays, C.W., Collins, A., Feron, E., Hobbs, K.L.: Natural motion-based trajectories for automatic spacecraft collision avoidance during proximity operations. In: 2021 IEEE Aerospace Conference (50100), pp. 1–12. IEEE (2021)
Schierman, J., Ward, D., Dutoi, B., et al.: Run-time verification and validation for safety-critical flight control systems. In: AIAA Paper 2008–6338, Proceedings of the AIAA Guidance, Navigation, and Control Conference, Honolulu, Hawaii, Aug., 2008 (2008)
Seto, D., Krogh, B., Sha, L., Chutinan, A.: The simplex architecture for safe online control system upgrades. In: American Control Conference (ACC) (1998)
Sha, L., et al.: Using simplicity to control complexity. IEEE Softw. 18(4), 20–28 (2001)
Shah, S., Dey, D., Lovett, C., Kapoor, A.: AirSim: high-fidelity visual and physical simulation for autonomous vehicles. In: Hutter, M., Siegwart, R. (eds.) Field and Service Robotics. SPAR, vol. 5, pp. 621–635. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-67361-5_40
Wadley, J., et al.: Development of an automatic aircraft collision avoidance system for fighter aircraft. In: AIAA Infotech@ Aerospace (I@ A) Conference, p. 4727 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Miller, K., Zeitler, C.K., Shen, W., Viswanathan, M., Mitra, S. (2023). RTAEval: A Framework for Evaluating Runtime Assurance Logic. In: André, É., Sun, J. (eds) Automated Technology for Verification and Analysis. ATVA 2023. Lecture Notes in Computer Science, vol 14216. Springer, Cham. https://doi.org/10.1007/978-3-031-45332-8_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-45332-8_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45331-1
Online ISBN: 978-3-031-45332-8
eBook Packages: Computer ScienceComputer Science (R0)