Skip to main content
SpringerLink
Log in

The black-box simplex architecture for runtime assurance of multi-agent CPS

  • Original Article
  • Published:
Innovations in Systems and Software Engineering Aims and scope Submit manuscript

Abstract

The Simplex Architecture is a runtime assurance framework where control authority may switch from an unverified and potentially unsafe advanced controller to a backup baseline controller in order to maintain the safety of an autonomous cyber-physical system. In this work, we show that runtime checks can replace the requirement to statically verify safety of the baseline controller. This is important as there are many powerful control techniques, such as model-predictive control and neural network controllers, that work well in practice but are difficult to statically verify. Since the method does not use internal information about the advanced or baseline controller, we call the approach the Black-Box Simplex Architecture. We prove the architecture is safe and present two case studies where (i) model-predictive control provides safe multi-robot coordination, and (ii) neural networks provably prevent collisions in groups of F-16 aircraft, despite the controllers occasionally outputting unsafe commands. We further show how to safely blend commands from the advanced and baseline controllers in multi-agent systems, reducing the performance impact when switching is necessary to preserve safety.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. A video of the simulation is available at https://youtu.be/bcVJBkGgnxA.

  2. A video of the simulation is available at https://youtu.be/qmk31jS6B2Y.

  3. https://github.com/sanazsheikhi/BlackBox-Simplex-Extension.

  4. https://youtu.be/6ZXjk8k-Xqs.

  5. https://youtu.be/Bhn0uqKCj7Q.

References

  1. Clark M, Koutsoukos X, Porter J, Kumar R, Pappas G, Sokolsky O, Lee I, Pike L (2013) A study on run time assurance for complex cyber physical systems. Technical report, Air Force Research Laboratory, Aerospace Systems Directorate

  2. Seto D, Krogh B, Sha L, Chutinan A (1998) The simplex architecture for safe online control system upgrades. In: Proceedings of the 1998 American control conference. ACC (IEEE Cat. No. 98CH36207), vol. 6. IEEE

  3. Sha L (2001) Using simplicity to control complexity. IEEE Softw 18(4):20–28. https://doi.org/10.1109/MS.2001.936213

    Article  Google Scholar 

  4. Desai A, Ghosh S, Seshia S.A, Shankar N, Tiwari A (2019) SOTER: a runtime assurance framework for programming safe robotics systems. In: 49th Annual IEEE/IFIP international conference on dependable systems and networks, DSN 2019, Portland, OR, USA, June 24–27, 2019

  5. Phan D, Yang J, Grosu R, Smolka SA, Stoller SD (2017) Collision avoidance for mobile robots with limited sensing and limited information about moving obstacles. Formal Methods Syst Design 51(1):62–86

    Article  Google Scholar 

  6. Schierman J, DeVore M.D, Richards N, Gandhi N, Cooper J, Horneman K.R, Stoller S, Smolka S (2015) Runtime assurance framework development for highly adaptive flight control systems. Report AD1010277, Defense Technical Information Center

  7. Mehmood U, Bak S, Smolka S.A, Stoller S.D (2021) Safe cps from unsafe controllers. In: Proceedings of the workshop on computation-aware algorithmic design for cyber-physical systems, pp 26–28

  8. Mehmood U, Sheikhi S, Bak S, Smolka S, Stoller S (2022) The black-box simplex architecture for runtime assurance of autonomous cps. In: NASA formal methods symposium

  9. Lin Q, Chen X, Khurana A, Dolan J (2020) Reachflow: an online safety assurance framework for waypoint-following of self-driving cars. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS)

  10. Bak S, Johnson T.T, Caccamo M, Sha L (2014) Real-time reachability for verified simplex design. In: 35th IEEE real-time systems symposium (RTSS 2014). IEEE Computer Society, Rome, Italy

  11. Althoff M, Dolan J.M (2014) Online verification of automated road vehicles using reachability analysis. IEEE Trans Robot 30(4)

  12. Phan D, Grosu R, Jansen N, Paoletti N, Smolka SA, Stoller SD (2020) Neural simplex architecture. In: NASA formal methods symposium (NFM 2020)

  13. Bak S, Chivukula D.K, Adekunle O, Sun M, Caccamo M, Sha L (2009) The system-level simplex architecture for improved real-time embedded system safety. In: 2009 15th IEEE real-time and embedded technology and applications symposium. IEEE, pp 99–107

  14. Kapinski J, Deshmukh J (2015) Discovering forward invariant sets for nonlinear dynamical systems. In: Interdisciplinary topics in applied mathematics, modeling and computational science, pp 259–264

  15. Murray RM, Li Z, Sastry SS, Sastry SS (1994) A mathematical introduction to robotic manipulation

  16. Khatib O (1986) Real-time obstacle avoidance for manipulators and mobile robots. In: Autonomous robot vehicles, pp 396–404

  17. Girard A (2005) Reachability of uncertain linear systems using zonotopes. In: International workshop on hybrid systems: computation and control. Springer

  18. Heidlauf P, Collins A, Bolender M, Bak S (2018) Verification challenges in f-16 ground collision avoidance and other automated maneuvers. In: 5th international workshop on applied verification of continuous and hybrid systems. EPiC Series in Computing, vol 54

  19. Stevens BL, Lewis FL, Johnson EN (2015) Aircraft control and simulation

  20. Kochenderfer MJ, Chryssanthacopoulos J (2011) Robust airborne collision avoidance through dynamic programming. Massachusetts Institute of Technology, Lincoln Laboratory, Project Report ATC-371 130

  21. Julian KD, Kochenderfer MJ, Owen MP (2019) Deep neural network compression for aircraft collision avoidance systems. J Guid Control Dyn 42(3):598–608

    Article  Google Scholar 

  22. Katz G, Barrett C, Dill DL, Julian K, Kochenderfer MJ (2017) Reluplex: an efficient SMT solver for verifying deep neural networks. In: International conference on computer aided verification. Springer, pp 97–117

  23. Marston M, Baca G (2015) ACAS-Xu initial self-separation flight tests. Technical report, NASA

  24. Bak S, Liu C, Johnson T (2021) The second international verification of neural networks competition (vnn-comp 2021): Summary and results. arXiv:2109.00498

  25. Bak S, Tran H-D, Hobbs K, Johnson TT (2020) Improved geometric path enumeration for verifying relu neural networks. In: Proceedings of the 32nd international conference on computer aided verification

  26. Chen X, Ábrahám E, Sankaranarayanan S (2013) Flow*: an analyzer for non-linear hybrid systems. In: International conference on computer aided verification. Springer, pp 258–263

  27. Schouwenaars T, Valenti M, Feron E, How J (2005) Implementation and flight test results of MILP-based UAV guidance. In: 2005 IEEE aerospace conference, pp 1–13

  28. Schouwenaars T (2006) Safe trajectory planning of autonomous vehicles. PhD thesis, Massachusetts Institute of Technology

  29. Alsterda JP, Brown M, Gerdes JC (2019) Contingency model predictive control for automated vehicles. In: 2019 American control conference (ACC), pp 717–722 . https://doi.org/10.23919/ACC.2019.8815260

  30. Magdici S, Althoff M (2016) Fail-safe motion planning of autonomous vehicles. In: 2016 IEEE 19th international conference on intelligent transportation systems (ITSC). IEEE, pp 452–458

  31. Schurmann B, Klischat M, Kochdumper N, Althoff M (2021) Formal safety net control using backward reachability analysis. IEEE Trans Autom Control

  32. Saint-Pierre P (1994) Approximation of the viability kernel. Appl Math Optim 29(2):187–209

    Article  MathSciNet  Google Scholar 

  33. Kaynama S, Maidens J, Oishi M, Mitchell IM, Dumont GA (2012) Computing the viability kernel using maximal reachable sets. In: Proceedings of the 15th ACM international conference on hybrid systems: computation and control, pp 55–64

  34. Maidens JN, Kaynama S, Mitchell IM, Oishi MM, Dumont GA (2013) Lagrangian methods for approximating the viability kernel in high-dimensional systems. Automatica 49(7):2017–2029

    Article  MathSciNet  Google Scholar 

  35. Phan D, Grosu R, Jansen N, Paoletti N, Smolka SA, Stoller SD (2020) Neural simplex architecture. In: NASA formal methods symposium (NFM 2020). Springer, pp 97–114

  36. Mashima D, Chen B, Zhou T, Rajendran R, Sikdar B (2018) Securing substations through command authentication using on-the-fly simulation of power system dynamics. In: IEEE international conference on communications, control, and computing technologies for smart grids

  37. Borrmann U, Wang L, Ames AD, Egerstedt M (2015) Control barrier certificates for safe swarm behavior. In: Egerstedt M, Wardi Y (eds) ADHS. IFAC-PapersOnLine, vol 48, pp 68–73

  38. Gurriet T, Mote M, Ames AD, Feron E (2018) An online approach to active set invariance. In: conference on decision and control. IEEE

  39. Gurriet T, Mote M, Singletary A, Feron E, Ames AD (2019) A scalable controlled set invariance framework with practical safety guarantees. In: 2019 IEEE 58th conference on decision and control (CDC). IEEE, pp 2046–2053

  40. Wang L, Han D, Egerstedt M (2018) Permissive barrier certificates for safe stabilization using sum-of-squares. In: 2018 Annual American control conference, ACC 2018. IEEE, pp 585–590

  41. Zhao H, Zeng X, Chen T, Liu Z (2020) Synthesizing barrier certificates using neural networks. In: Proceedings of the 23rd international conference on hybrid systems: computation and control. HSCC ’20. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3365365.3382222

  42. Ames AD, Coogan S, Egerstedt M, Notomista G, Sreenath K, Tabuada P (2019) Control barrier functions: theory and applications. In: 18th European control conference, ECC 2019, Naples, Italy. IEEE, pp 3420–3431

  43. Mehmood U, Roy S, Damare A, Grosu R, Smolka SA, Stoller SD (2023) A distributed simplex architecture for multi-agent systems. J Syst Architect 134:102784. https://doi.org/10.1016/j.sysarc.2022.102784

    Article  Google Scholar 

  44. Raju D, Bharadwaj S, Djeumou F, Topcu U (2021) Online synthesis for runtime enforcement of safety in multiagent systems. IEEE Trans Control Netw Syst 8(2):621–632. https://doi.org/10.1109/TCNS.2021.3061900

    Article  MathSciNet  Google Scholar 

  45. Engelmann DC, Ferrando A, Panisson AR, Ancona D, Bordini RH, Mascardi V (2022) RV4jaca-runtime verification for multi-agent systems. Electron Proc Theor Comput Sci 362:23–36. https://doi.org/10.4204/eptcs.362.5

    Article  Google Scholar 

  46. Schneider FB (2000) Enforceable security policies 3(1):30–50. https://doi.org/10.1145/353323.353382

  47. Bauer L, Ligatti J, Walker D (2002) More enforceable security policies

  48. Falcone Y, Mounier L, Fernandez J-C, Richier J-L (2011) Runtime enforcement monitors: composition, synthesis, and enforcement abilities. Formal Methods Syst Design. https://doi.org/10.1007/s10703-011-0114-4

    Article  Google Scholar 

  49. Pinisetty S, Preoteasa V, Tripakis S, Jéron T, Falcone Y, Marchand H (2016) Predictive runtime enforcement, pp 1628–1633 . https://doi.org/10.1145/2851613.2851827

  50. Rania Taleb RK, Hallé S (2022) A modular runtime enforcement model using multi-traces. Foundations and Practice of Security Lecture Notes in Computer Science, pp 283–302

Download references

Acknowledgements

This material is based upon work supported by the Air Force Office of Scientific Research and the Office of Naval Research under award numbers FA9550-19-1-0288, FA9550-21-1-0121, FA9550-23-1-0066, N00014-22-1-2156, and N000142112719, and the National Science Foundation under Award No. CNS-2237229, ITE-2040599, ITE-2134840, CCF-1954837, CCF-1918225 and CPS-1446832. % Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force or the United States Navy.

Author information

Authors and Affiliations

  1. Department of Computer Science, Stony Brook University, Stony Brook, NY, USA

    Sanaz Sheikhi, Stanley Bak, Scott A. Smolka & Scott D. Stoller

  2. Department of Computer Science, Information Technology University, Lahore, Pakistan

    Usama Mehmood

Authors
  1. Sanaz Sheikhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Usama Mehmood
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stanley Bak
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Scott A. Smolka
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Scott D. Stoller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Stanley Bak, Scott A. Smolka or Scott D. Stoller.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sheikhi, S., Mehmood, U., Bak, S. et al. The black-box simplex architecture for runtime assurance of multi-agent CPS. Innovations Syst Softw Eng (2024). https://doi.org/10.1007/s11334-024-00553-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11334-024-00553-6

Keywords

Search

Navigation