Abstract
The Simplex Architecture is a runtime assurance framework where control authority may switch from an unverified and potentially unsafe advanced controller to a backup baseline controller in order to maintain the safety of an autonomous cyber-physical system. In this work, we show that runtime checks can replace the requirement to statically verify safety of the baseline controller. This is important as there are many powerful control techniques, such as model-predictive control and neural network controllers, that work well in practice but are difficult to statically verify. Since the method does not use internal information about the advanced or baseline controller, we call the approach the Black-Box Simplex Architecture. We prove the architecture is safe and present two case studies where (i) model-predictive control provides safe multi-robot coordination, and (ii) neural networks provably prevent collisions in groups of F-16 aircraft, despite the controllers occasionally outputting unsafe commands.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
A video of the simulation is available at https://youtu.be/bcVJBkGgnxA.
- 2.
A video of the simulation is available at https://youtu.be/qmk31jS6B2Y.
- 3.
- 4.
References
Alsterda, J.P., Brown, M., Gerdes, J.C.: Contingency model predictive control for automated vehicles. In: 2019 American Control Conference (ACC), pp. 717–722 (2019). https://doi.org/10.23919/ACC.2019.8815260
Althoff, M., Dolan, J.M.: Online verification of automated road vehicles using reachability analysis. IEEE Trans. Robot. 30(4) (2014)
Bak, S., Chivukula, D.K., Adekunle, O., Sun, M., Caccamo, M., Sha, L.: The system-level simplex architecture for improved real-time embedded system safety. In: 2009 15th IEEE Real-Time and Embedded Technology and Applications Symposium, pp. 99–107. IEEE (2009)
Bak, S., Johnson, T.T., Caccamo, M., Sha, L.: Real-time reachability for verified simplex design. In: 35th IEEE Real-Time Systems Symposium (RTSS 2014). IEEE Computer Society, Rome, December 2014
Bak, S., Liu, C., Johnson, T.: The second international verification of neural networks competition (VNN-COMP 2021): summary and results. arXiv preprint arXiv:2109.00498 (2021)
Bak, S., Tran, H.D., Hobbs, K., Johnson, T.T.: Improved geometric path enumeration for verifying Relu neural networks. In: Proceedings of the 32nd International Conference on Computer Aided Verification (2020)
Borrmann, U., Wang, L., Ames, A.D., Egerstedt, M.: Control barrier certificates for safe swarm behavior. In: Egerstedt, M., Wardi, Y. (eds.) ADHS. IFAC-PapersOnLine, vol. 48, pp. 68–73. Elsevier, Amsterdam (2015)
Chen, X., Ábrahám, E., Sankaranarayanan, S.: Flow*: an analyzer for non-linear hybrid systems. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 258–263. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39799-8_18
Clark, M., et al.: A study on run time assurance for complex cyber physical systems. Technical report, Air Force Research Laboratory, Aerospace Systems Directorate (2013)
Desai, A., Ghosh, S., Seshia, S.A., Shankar, N., Tiwari, A.: SOTER: a runtime assurance framework for programming safe robotics systems. In: 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019, Portland, OR, USA, 24–27 June 2019. IEEE (2019)
Girard, A.: Reachability of uncertain linear systems using zonotopes. In: Morari, M., Thiele, L. (eds.) HSCC 2005. LNCS, vol. 3414, pp. 291–305. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-31954-2_19
Gurriet, T., Mote, M., Ames, A.D., Feron, E.: An online approach to active set invariance. In: Conference on Decision and Control. IEEE (2018)
Gurriet, T., Mote, M., Singletary, A., Feron, E., Ames, A.D.: A scalable controlled set invariance framework with practical safety guarantees. In: 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 2046–2053. IEEE (2019)
Heidlauf, P., Collins, A., Bolender, M., Bak, S.: Verification challenges in f-16 ground collision avoidance and other automated maneuvers. In: 5th International Workshop on Applied Verification of Continuous and Hybrid Systems. EPiC Series in Computing, EasyChair (2018)
Julian, K.D., Kochenderfer, M.J., Owen, M.P.: Deep neural network compression for aircraft collision avoidance systems. J. Guid. Control. Dyn. 42(3), 598–608 (2019)
Kapinski, J., Deshmukh, J.: Discovering forward invariant sets for nonlinear dynamical systems. In: Cojocaru, M.G., Kotsireas, I.S., Makarov, R.N., Melnik, R.V.N., Shodiev, H. (eds.) Interdisciplinary Topics in Applied Mathematics, Modeling and Computational Science. SPMS, vol. 117, pp. 259–264. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-12307-3_37
Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_5
Kaynama, S., Maidens, J., Oishi, M., Mitchell, I.M., Dumont, G.A.: Computing the viability kernel using maximal reachable sets. In: Proceedings of the 15th ACM International Conference on Hybrid Systems: Computation and Control, pp. 55–64 (2012)
Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. In: Cox, I.J., Wilfong, G.T. (eds.) Autonomous Robot Vehicles, pp. 396–404. Springer, New York (1986). https://doi.org/10.1007/978-1-4613-8997-2_29
Kochenderfer, M.J., Chryssanthacopoulos, J.: Robust airborne collision avoidance through dynamic programming. Project Report ATC-371 130, Lincoln Laboratory, Massachusetts Institute of Technology (2011)
Lin, Q., Chen, X., Khurana, A., Dolan, J.: ReachFlow: an online safety assurance framework for waypoint-following of self-driving cars. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2020)
Magdici, S., Althoff, M.: Fail-safe motion planning of autonomous vehicles. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 452–458. IEEE (2016)
Maidens, J.N., Kaynama, S., Mitchell, I.M., Oishi, M.M., Dumont, G.A.: Lagrangian methods for approximating the viability kernel in high-dimensional systems. Automatica 49(7), 2017–2029 (2013)
Marston, M., Baca, G.: ACAS-Xu initial self-separation flight tests. Technical report, NASA (2015)
Mashima, D., Chen, B., Zhou, T., Rajendran, R., Sikdar, B.: Securing substations through command authentication using on-the-fly simulation of power system dynamics. In: IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (2018)
Mehmood, U., Bak, S., Smolka, S.A., Stoller, S.D.: Safe cps from unsafe controllers. In: Proceedings of the Workshop on Computation-Aware Algorithmic Design for Cyber-Physical Systems, pp. 26–28 (2021)
Murray, R.M., Li, Z., Sastry, S.S., Sastry, S.S.: A Mathematical Introduction to Robotic Manipulation. CRC Press, Boca Raton (1994)
Lee, R., Jha, S., Mavridou, A., Giannakopoulou, D. (eds.): NFM 2020. LNCS, vol. 12229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55754-6
Phan, D.T., Grosu, R., Jansen, N., Paoletti, N., Smolka, S.A., Stoller, S.D.: Neural simplex architecture. In: Lee, R., Jha, S., Mavridou, A., Giannakopoulou, D. (eds.) NFM 2020. LNCS, vol. 12229, pp. 97–114. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55754-6_6
Phan, D., Yang, J., Grosu, R., Smolka, S.A., Stoller, S.D.: Collision avoidance for mobile robots with limited sensing and limited information about moving obstacles. Formal Methods Syst. Des. 51(1), 62–86 (2017). https://doi.org/10.1007/s10703-016-0265-4
Saint-Pierre, P.: Approximation of the viability kernel. Appl. Math. Optim. 29(2), 187–209 (1994)
Schierman, J., et al.: Runtime assurance framework development for highly adaptive flight control systems. Report AD1010277, Defense Technical Information Center (2015)
Schouwenaars, T., Valenti, M., Feron, E., How, J.: Implementation and flight test results of MILP-based UAV guidance. In: 2005 IEEE Aerospace Conference, pp. 1–13 (2005)
Schouwenaars, T.: Safe trajectory planning of autonomous vehicles. Ph.D. thesis, Massachusetts Institute of Technology (2006)
Schurmann, B., Klischat, M., Kochdumper, N., Althoff, M.: Formal safety net control using backward reachability analysis. IEEE Trans. Autom. Control (2021)
Seto, D., Krogh, B., Sha, L., Chutinan, A.: The simplex architecture for safe online control system upgrades. In: Proceedings of the 1998 American Control Conference. ACC (IEEE Cat. No. 98CH36207), vol. 6. IEEE (1998)
Sha, L.: Using simplicity to control complexity. IEEE Softw. 18(4), 20–28 (2001). https://doi.org/10.1109/MS.2001.936213
Stevens, B.L., Lewis, F.L., Johnson, E.N.: Aircraft Control and Simulation. Wiley, New York (2015)
Acknowledgement
This material is based upon work supported by National Science Foundation (NSF) under grant numbers OIA-2134840, OIA-2040599, CCF-1918225, CCF-1954837 and CPS-1446832, the Office of Naval Research (ONR) under grants N000142112719 and N000142212156, and the Air Force Office of Scientific Research (AFOSR) under award numbers FA9550-19-1-0288, FA9550-21-1-0121, FA9550-22-1-0450. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF, United States Air Force or the United States Navy. An early version of this work was presented in the CAADCPS 2021 workshop under the title “Safe CPS from Unsafe Controllers” [26].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Mehmood, U., Sheikhi, S., Bak, S., Smolka, S.A., Stoller, S.D. (2022). The Black-Box Simplex Architecture for Runtime Assurance of Autonomous CPS. In: Deshmukh, J.V., Havelund, K., Perez, I. (eds) NASA Formal Methods. NFM 2022. Lecture Notes in Computer Science, vol 13260. Springer, Cham. https://doi.org/10.1007/978-3-031-06773-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-06773-0_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-06772-3
Online ISBN: 978-3-031-06773-0
eBook Packages: Computer ScienceComputer Science (R0)