Skip to main content
SpringerLink
Log in

Reaching Out Towards Fully Verified Autonomous Systems

  • Conference paper
  • First Online:
Reachability Problems (RP 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11674))

Included in the following conference series:

Abstract

Autonomous systems such as “self-driving” vehicles and closed-loop medical devices increasingly rely on learning-enabled components such as neural networks to perform safety critical perception and control tasks. As a result, the problem of verifying that these systems operate correctly is of the utmost importance. We will briefly examine the role of neural networks in the design and implementation of autonomous systems, and how various verification approaches can contribute towards engineering verified autonomous systems. In doing so, we examine promising initial solutions that have been proposed over the past three years and the big challenges that remain to be tackled.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abadi, M., Agarwal, A., Barham, P., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/

  2. Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., Topcu, U.: Safe reinforcement learning via shielding (2018). https://aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17211

  3. Shih, A., Darwiche, A., Choi, A.: Verifying binarized neural networks by local automaton learning (2019). http://reasoning.cs.ucla.edu/fetch.php?id=193&type=pdf

  4. Annpureddy, Y., Liu, C., Fainekos, G., Sankaranarayanan, S.: S-TaLiRo: a tool for temporal logic falsification for hybrid systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 254–257. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19835-9_21

    Chapter  MATH  Google Scholar 

  5. Bojarski, M., et al.: End to end learning for self-driving cars. CoRR abs/1604.07316 (2016). http://arxiv.org/abs/1604.07316

  6. Chen, X., Ábrahám, E., Sankaranarayanan, S.: Flow*: an analyzer for non-linear hybrid systems. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 258–263. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39799-8_18

    Chapter  Google Scholar 

  7. Chen, X., Sankaranarayanan, S.: Model-predictive real-time monitoring of linear systems. In: IEEE Real-Time Systems Symposium (RTSS), pp. 297–306. IEEE Press (2017)

    Google Scholar 

  8. Cheng, C., Nührenberg, G., Ruess, H.: Maximum resilience of artificial neural networks. CoRR abs/1705.01040 (2017). http://arxiv.org/abs/1705.01040

  9. Cheng, C., Nührenberg, G., Ruess, H.: Verification of binarized neural networks. CoRR abs/1710.03107 (2017). http://arxiv.org/abs/1710.03107

  10. Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: ACM Principles of Programming Languages, pp. 238–252 (1977)

    Google Scholar 

  11. Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Sig. Syst. 2, 303–314 (1989)

    Article  MathSciNet  Google Scholar 

  12. Dreossi, T., Donzé, A., Seshia, S.A.: Compositional falsification of cyber-physical systems with machine learning components. In: Barrett, C., Davies, M., Kahsai, T. (eds.) NFM 2017. LNCS, vol. 10227, pp. 357–372. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57288-8_26

    Chapter  Google Scholar 

  13. Dutta, S., Chen, X., Sankaranarayanan, S.: Reachability analysis for neural feedback systems using regressive polynomial rule inference. In: Proceedings of the Hybrid Systems: Computation and Control (HSCC), HSCC 2019, pp. 157–168. ACM, New York (2019)

    Google Scholar 

  14. Dutta, S., Kushner, T., Sankaranarayanan, S.: Robust data-driven control of artificial pancreas systems using neural networks. In: Češka, M., Šafránek, D. (eds.) CMSB 2018. LNCS, vol. 11095, pp. 183–202. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99429-1_11

    Chapter  MATH  Google Scholar 

  15. Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: Proceedings of the ACM Programming Language Design and Implementation (PLDI), pp. 63–78 (2019)

    Google Scholar 

  16. Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.: Ai2: safety and robustness certification of neural networks with abstract interpretation. In: 2018 IEEE Symposium on Security and Privacy (SP), pp. 3–18, May 2018

    Google Scholar 

  17. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The Kitti vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361, June 2012

    Google Scholar 

  18. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org

  19. Hashimoto, D.A., Rosman, G., Rus, D., Meireles, O.: Artificial intelligence in surgery: promises and perils. Ann. Surg. 268, 70–76 (2018)

    Article  Google Scholar 

  20. Huang, C., Fan, J., Li, W., Chen, X., Zhu, Q.: Reachnn: reachability analysis of neural-network controlled systems. CoRR abs/1906.10654 (2019). http://arxiv.org/abs/1906.10654

  21. Ivanov, R., Weimer, J., Alur, R., Pappas, G.J., Lee, I.: Verisig: verifying safety properties of hybrid systems with neural network controllers. In: Proceedings of the Hybrid Systems: Computation and Control (HSCC), HSCC 2019, pp. 169–178. ACM, New York (2019)

    Google Scholar 

  22. LeCun, Y., Kavukcuoglu, K., Farabet, C.: Convolutional networks and applications in vision. In: Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pp. 253–256, May 2010. https://doi.org/10.1109/ISCAS.2010.5537907

  23. Narodytska, N., Kasiviswanathan, S.P., Ryzhyk, L., Sagiv, M., Walsh, T.: Verifying properties of binarized deep neural networks. CoRR abs/1709.06662 (2017). http://arxiv.org/abs/1709.06662

  24. Nielson, F., Nielson, H.R., Hankin, C.: Principles of Program Analysis. Springer, Heidelberg (1999). https://doi.org/10.1007/978-3-662-03811-6

    Book  MATH  Google Scholar 

  25. Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS Workshop on Automatic Differentiation (2017). https://openreview.net/forum?id=BJJsrmfCZ

  26. Prajna, S., Jadbabaie, A.: Safety verification using barrier certificates. In: Proceedings of the HSCC 2004, vol. 2993, pp. 477–492 (2004)

    Google Scholar 

  27. Sha, L.: Using simplicity to control complexity. IEEE Softw. 18(4), 20–28 (2001)

    Article  Google Scholar 

  28. Sun, X., Khedr, H., Shoukry, Y.: Formal verification of neural network controlled autonomous systems. In: Proceedings of the Hybrid Systems: Computation and Control (HSCC), HSCC 2019, pp. 147–156. ACM, New York (2019)

    Google Scholar 

  29. Tuncali, C.E., Fainekos, G., Ito, H., Kapinski, J.: Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: 2018 IEEE Intelligent Vehicles Symposium, pp. 1555–1562 (2018)

    Google Scholar 

  30. Tuncali, C.E., Kapinski, J., Ito, H., Deshmukh, J.V.: Reasoning about safety of learning-enabled components in autonomous cyber-physical systems. In: Proceedings of the Design Automation Conference, DAC 2018, pp. 30:1–30:6 (2018)

    Google Scholar 

  31. U.S Food and Drug Administration: Computer-assisted surgical systems (2019). https://www.fda.gov/medical-devices/surgery-devices/computer-assisted-surgical-systems. Accessed July 2019

  32. Wang, S., Chen, Y., Abdou, A., Jana, S.: Mixtrain: scalable training of formally robust neural networks. CoRR abs/1811.02625 (2018). http://arxiv.org/abs/1811.02625

  33. Wang, S., Pei, K., Whitehouse, J., Yang, J., Jana, S.: Formal security analysis of neural networks using symbolic intervals. CoRR abs/1804.10829 (2018). http://arxiv.org/abs/1804.10829

  34. Wong, E., Kolter, J.Z.: Provable defenses against adversarial examples via the convex outer adversarial polytope. In: Proceedings of the International Conference on Machine Learning, ICML, pp. 5283–5292 (2018). http://proceedings.mlr.press/v80/wong18a.html

  35. Xiang, W., Tran, H., Johnson, T.T.: Reachable set computation and safety verification for neural networks with relu activations. CoRR abs/1712.08163 (2017). http://arxiv.org/abs/1712.08163

  36. Xiang, W., Tran, H.D., Johnson, T.T.: Reachable set computation and safety verification for neural networks with relu activations (2107). https://arxiv.org/pdf/1712.08163.pdf. Posted on arxiv December 2017

  37. Xiang, W., Tran, H.D., Rosenfeld, J.A., Johnson, T.T.: Reachable set estimation and verification for a class of piecewise linear systems with neural network controllers (2018). To Appear in the American Control Conference (ACC), invited session on Formal Methods in Controller Synthesis

    Google Scholar 

  38. Yaghoubi, S., Fainekos, G.: Gray-box adversarial testing for control systems with machine learning components. In: Proceedings of Hybrid Systems: Computation and Control, pp. 179–184 (2019)

    Google Scholar 

  39. Yoon, H., Chou, Y., Chen, X., Frew, E., Sankaranarayanan, S.: Predictive runtime monitoring for linear stochastic systems and applications to geofence enforcement for UAVs (2019). In: Proceedings of the Runtime Verification 2019, October 2019 (to appear)

    Google Scholar 

  40. Zhu, H., Xiong, Z., Magill, S., Jagannathan, S.: An inductive synthesis framework for verifiable reinforcement learning. In: ACM Programming Language Design and Implementation (PLDI), pp. 686–701 (2019)

    Google Scholar 

Download references

Acknowledgments

This work was supported in part by the Air Force Research Laboratory (AFRL) and by the US NSF under Award # 1646556.

Author information

Authors and Affiliations

  1. University of Colorado, Boulder, USA

    Sriram Sankaranarayanan & Souradeep Dutta

  2. Ecole Polytechnique, Institut Polytechnique de Paris, Palaiseau, France

    Sergio Mover

Authors
  1. Sriram Sankaranarayanan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Souradeep Dutta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sergio Mover
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sriram Sankaranarayanan .

Editor information

Editors and Affiliations

  1. Université Libre de Bruxelles, Brussels, Belgium

    Emmanuel Filiot

  2. Université Catholique de Louvain, Louvain-la-Neuve, Belgium

    Raphaël Jungers

  3. University of Liverpool, Liverpool, UK

    Igor Potapov

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sankaranarayanan, S., Dutta, S., Mover, S. (2019). Reaching Out Towards Fully Verified Autonomous Systems. In: Filiot, E., Jungers, R., Potapov, I. (eds) Reachability Problems. RP 2019. Lecture Notes in Computer Science(), vol 11674. Springer, Cham. https://doi.org/10.1007/978-3-030-30806-3_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30806-3_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30805-6

  • Online ISBN: 978-3-030-30806-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation