Abstract
Self-driving cars over the last decade have achieved significant progress like driving millions of miles without any human intervention. However, behavioral safety in applying deep-neural-network-based (DNN based) systems for self-driving cars could not be guaranteed. Several real-world accidents involving self-driving cars have already happened, some of which have led to fatal collisions. In this paper, we present a novel and automated technique for verifying steering angle safety for self-driving cars. The technique is based on deep learning verification (DLV), which is an automated verification framework for safety of image classification neural networks. We extend DLV by leveraging neuron coverage and slack relationship to solve the judgement problem of predicted behaviors, and thus, to achieve verification of steering angle safety for self-driving cars. We evaluate our technique on the NVIDIA’s end-to-end self-driving architecture, which is a crucial ingredient in many modern self-driving cars. Experimental results show that our technique can successfully find adversarial misclassifications (i.e., incorrect steering decisions) within given regions if they exist. Therefore, we can achieve safety verification (if no misclassification is found for all DNN layers, in which case the network can be said to be stable or reliable w.r.t. steering decisions) or falsification (in which case the adversarial examples can be used to fine-tune the network).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Althoff, M., Dolan John, M.: Online verification of automated road vehicles using reachability analysis. IEEE Trans Robot 30(4), 903–918 (2014)
Bastani O, Ioannou Y, Lampropoulos L, Vytiniotis D, Nori AV, Criminisi A (2016) Measuring neural net robustness with constraints. In: Advances in neural information processing systems 29: annual conference on neural information processing systems 2016, December 5–10, 2016, Barcelona, Spain, pp 2613–2621
Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. In: Advances in neural information processing systems 19, proceedings of the twentieth annual conference on neural information processing systems, Vancouver, British Columbia, Canada, December 4–7, 2006, pp 153–160
Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J, Zhang X, Zhao J, Zieba K (2016) End to end learning for self-driving cars. CoRR, arXiv:1604.07316
Chauffeur model (2016)
The numbers dont lie: self-driving cars are getting good (2017)
Hinton GE, Rumelhart DE, Williams RJ (1998) Learning representations by back-propagating errors. In: Cognitive modeling, 1998
Epoch model (2016)
Fawzi A, Fawzi O, Frossard P (2015) Analysis of classifiers' robustness to adversarial perturbations. CoRR, arXiv:1502.02590
Feth P, Schneider D, Adler R (2017) A conceptual safety supervisor definition and evaluation framework for autonomous systems. In: Computer safety, reliability, and security—36th international conference, SAFECOMP 2017, Trento, Italy, September 13–15, 2017, proceedings, pp 135–148
Goodfellow, I.J., Bengio, Y., Courville, A.C.: Deep learning. MIT Press, Berlin, Adaptive computation and machine learning (2016)
Google's self-driving car caused its first crash (2016)
Hauke J, Kossowski T (2011) Comparison of values of Pearsons and Spearmans correlation coefficients on the same sets of data. In: Quaestiones geographicae, p 87
Huang X, Kwiatkowska M, Wang S, Wu M (2017) Safety verification of deep neural networks. In: Computer aided verification—29th international conference, CAV 2017, Heidelberg, Germany, July 24–28, 2017, proceedings, part I, pp 3–29
Huang X
Katz G, Barrett CW, Dill DL, Julian K, Kochenderfer MJ (2017) Reluplex: an efficient SMT solver for verifying deep neural networks. In: Computer aided verification—29th international conference, CAV 2017, Heidelberg, Germany, July 24–28, 2017, proceedings, part I, pp 97–117
Katz G, Barrett CW, Dill DL, Julian K, Kochenderfer MJ (2017) Reluplex: an efficient SMT solver for verifying deep neural networks. CoRR, arXiv:1702.01135
Kane A, Chowdhury O, Datta A, Koopman P (2015) A case study on runtime monitoring of an autonomous research vehicle (ARV) system. In: Runtime verification–6th international conference, RV 2015 Vienna, Austria, September 22–25, 2015. Proceedings, pp 102–117
Keras (2019). https://keras.io
Kalra N, Paddock SM (2016) Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability?. In: Transportation research part A: policy and practice, vol 94, p 182C193
Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), June 21–24, 2010, Haifa, Israel, pp 807–814
Naumann M, Königshof H, Stiller C (2019) Provably safe and smooth lane changes in mixed trafic. In: 2019 IEEE intelligent transportation systems conference, ITSC 2019, Auckland, New Zealand, October 27–30, 2019, pp 1832–1837
Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th symposium on operating systems principles, Shanghai, China, October 28–31, 2017, pp 1–18
Pek C, Koschi M, Althoff M (2019) An online verification framework for motion planning of self-driving vehicles with safety guarantees. In: AAET
Papernot N, McDaniel PD, Goodfellow IJ, Jha S, Celik ZB, Swami A (2016) Practical black-box attacks against deep learning systems using adversarial examples. CoRR, arXiv:1602.02697
Pulina, L., Tacchella, A.: Challenging SMT solvers to verify neural networks. AI Commun 25(2), 117–135 (2012)
Pek C, Zahn P, Althoff M (2017) Verifying the safety of lane change maneuvers of self-driving vehicles based on formalized traffic rules. In: IEEE intelligent vehicles symposium, IV 2017, Los Angeles, CA, USA, June 11–14, 2017, pp 1477–1483
Rambo model (2016)
Rambo model (2017). https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/rambo
Roohi N, Kaur R, Weimer J, Sokolsky O, Lee I (2018) Self-driving vehicle verification towards a benchmark. CoRR, arXiv:1806.08810
Stahl T, Eicher M, Betz J, Diermeyer F (2020) Online verification concept for autonomous vehicles—illustrative study for a trajectory planning module. CoRR, arXiv:2005.07740
Schürmann B, Hess D, Eilbrecht J, Stursberg O, Köster F, Althoff M (2017) Ensuring drivability of planned motions using formal methods. In: 20th IEEE international conference on intelligent transportation systems, ITSC 2017, Yokohama, Japan, October 16–19, 2017, pp 1–8
Rishi K, Samer H, Chris R (2015) Using convolutional neural networks for image recognition. Technical report, 2015
Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 72C101
Shalev-Shwartz S, Shammah S, Shashua A (2017) On a formal model of safe and scalable self-driving cars. CoRR, arXiv:1708.06374
SullyChen. Driving dataset
Safety and trustworthiness of deep neural networks: a survey. CoRR, arXiv:1812.08342, 2018. Withdrawn
Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow IJ, Fergus R (2014) Intriguing properties of neural networks. In: 2nd international conference on learning representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, conference track proceedings
Tesla says vehicle in deadly crash was on autopilot (2018)
Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th international conference on software engineering, ICSE 2018, Gothenburg, Sweden, May 27–June 03, 2018, pp 303–314
Cheung SC, Chen TY, Yiu SM (1998) Metamorphic testing: a new approach for generating next test cases. Technical report, technical report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong
Fatal car crash involving a self-driving uber shows there are major flaws in the software, hardware, and testing procedures involving autonomous vehicles (2018)
Z3 (2019). http://rise4fun.com/z3
Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018) Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ASE 2018, Montpellier, France, September 3–7, 2018, pp 132–142
Author information
Authors and Affiliations
Corresponding author
Additional information
Zhiming Liu, Xiaoping Chen, Ji Wang and Jim Woodcock
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wu, H., Lv, D., Cui, T. et al. SDLV: Verification of Steering Angle Safety for Self-Driving Cars. Form Asp Comp 33, 325–341 (2021). https://doi.org/10.1007/s00165-021-00539-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00165-021-00539-2