Abstract
Machine Learning components in safety-critical applications can perform some complex tasks that would be unfeasible otherwise. However, they are also a weak point concerning safety assurance. An aspect requiring study is how the interactions between machine-learning components and other non-ML components evolve with training of the former. It is theoretically possible that learning by Neural Networks may reduce the effectiveness of error checkers or safety monitors, creating a major complication for safety assurance. We present an initial exploration of this problem focused on automated driving, where machine learning is heavily used. We simulated operational testing of a standard vehicle architecture, where a machine learning-based Controller is responsible for driving the vehicle and a separate Safety Monitor is provided to detect hazardous situations and trigger emergency action to avoid accidents. Among the results, we observed that indeed improving the Controller could make the Safety Monitor less effective; it is even possible for a training increment to make the Controller’s own behaviour safer but the vehicle’s less safe. We discuss implications for practice and for research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
WAYMO: Technology. https://waymo.com/tech/. Accessed 23 Dec 2021
NVIDIA: Training AI for Self-Driving Vehicles: the challenge of scale. https://developer.nvidia.com. Accessed 23 Dec 2021
Drago Anguelov (Waymo): MIT Self-Driving Cars (2019)
WAYMO: Waymo Safety Report (2021)
UNECE: UN Regulation on Advanced Emergency Braking Systems for cars to significantly reduce crashes (2019)
EU: Road safety: commission welcomes agreement on new EU rules to help save lives (2019)
American Safety Council – Should Autonomous Emergency Braking be Mandatory?
Popov, P., Strigini, L.: Assessing asymmetric fault-tolerant software. ISSRE, IEEE (2010)
Zhao, X. et al.: Assessing safety-critical systems from operational testing: a study on autonomous vehicles. Inform. Software Technol. 128, 106393 (2020)
Littlewood, B., Strigini, L.: Validation of ultrahigh dependability for software-based systems. Commun. ACM 36, 69–80 (1993)
Butler, R.W., Finelli, G.B.: The infeasibility of quantifying the reliability of life-critical real-time software. IEEE Trans. Software Eng. 19(1), 3–12 (1993)
Kalra, N., Paddock, S.M.: Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice (2016)
Koopman, P., Wagner, M.: Autonomous vehicle safety: an interdisciplinary challenge. IEEE Intell. Transp. Syst. Magaz. 9, 90–96 (2017)
Varshney, K.R.: Engineering safety in machine learning. IEEE ITA (2016)
Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CVPR, IEEE, pp. 427–436 (2015)
Zhao, D., et al.: Autonomous driving simulation for unmanned vehicles. In: IEEE Winter Conference on Applications of Computer Vision, pp. 185–190 (2015)
Baltodano, S., et al.: The RRADS platform: a real road autonomous driving simulator. In: Proceedings of AUTOUI, pp. 281–288 (2015)
Osiński, B., et al.: Simulation-based reinforcement learning for real-world autonomous driving. IEEE ICRA, pp. 6411–6418 (2020)
Koopman, P., Wagner, M.: Challenges in autonomous vehicle testing and validation. SAE Int. J. Transp. Saf. 4(1), 15–24 (2016)
Grigorescu, S., et al.: A survey of deep learning techniques for autonomous driving. J. Field Robot. 37(3), 362–386 (2020)
Dosovitskiy, A., et al.: CARLA: an open urban driving simulator. CoRL, pp. 1–16 (2017)
Lillicrap, T.P., et al.: Continuous Control with Reinforcement Learning. arXiv:150902971 (2015)
Caspi, I., Leibovich, G., Novik, G., Endrawis, S.: Reinforcement Learning Coach (2017)
Rusu, R.B., Cousins, S.: 3d is here: Point cloud library (pcl). IEEE ICRA, pp. 1–4 (2011)
Bozkurt, E.: LidarObstacleDetection (2019). https://github.com/enginBozkurt/
Greengard, S.: Gaming machine learning. Commun. ACM 60.12 (2017)
Singh, N., et al.: Facial recognition using deep learning. In: Jain, V., Chaudhary, G., Taplamacioglu, M., Agarwal, M. (eds.) Advances in Data Sciences, Security and Applications, LNEE, vol. 612, pp. 375–382. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-0372-6_30
Rao, Q., Jelena F.: Deep learning for self-driving cars: Chances and challenges. SEFAIS 2018
Ravi, M., Kantheti, S.C.: Application of artificial intelligence in healthcare: chances and challenges. Curr. J. Appl. Sci. Technol. (2021)
Hoang, D.-T., Kang, H.-J.: A survey on deep learning based bearing fault diagnosis. Neurocomputing 335, 327–335 (2019)
Jesse, L., et al.: Towards fully autonomous driving: Systems and algorithms. In: 2011 IEEE Intelligent Vehicles Symposium (IV). IEEE (2011)
Koorosh, A., et al.: SafeML: Safety monitoring of machine learning classifiers through statistical difference measures. In: Zeller, M., Höfig, K. (eds.) IMBSA, LNPSE, vol. 12297. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58920-2_13
Kurd, Z., Kelly, T., Austin, J.: Developing artificial neural networks for safety critical systems. Neural Comput. Appl. 16(1), 11–19 (2007)
Randy, G., et al.: Explainable AI: the new 42? In: Holzinger, A., Kieseberg, P., Tjoa, A., Weippl, E. (eds.) CD-MAKE, LNISA, vol. 11015. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99740-7_21
Cheng, C.-H.: Safety-aware hardening of 3D object detection neural network systems. In: Casimiro, A., Ortmeier, F., Bitsch, F., Ferreira, P. (eds.) SAFECOMP, LNPSE, vol. 12234. Springer, Cham, 2020. https://doi.org/10.1007/978-3-030-54549-9_14
Koopman, P., et al.: Credible autonomy safety argumentation. SCSC, UK (2019)
Gauerhof, L., Munk, P., Burton, S.: Structuring validation targets of a machine learning function applied to automated driving. In: Gallina, B., Skavhaug, A., Bitsch, F. (eds.) SAFECOMP 2018. LNCS, vol. 11093, pp. 45–58. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99130-6_4
Huang, X., et al.: A survey of safety and trustworthiness of deep neural networks: verification, testing, adversarial attack and defence, and interpretability. Comput. Sci. Rev. 37, 100270 (2020)
Acknowledgements
Strigini’s work was supported in part by ICRI-SAVe, the Intel Collaborative Research Institute on Safe Automated Vehicles. The authors are grateful to Peter Bishop for his insightful comments on the results.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Terrosi, F., Strigini, L., Bondavalli, A. (2022). Impact of Machine Learning on Safety Monitors. In: Trapp, M., Saglietti, F., Spisländer, M., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2022. Lecture Notes in Computer Science, vol 13414. Springer, Cham. https://doi.org/10.1007/978-3-031-14835-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-14835-4_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14834-7
Online ISBN: 978-3-031-14835-4
eBook Packages: Computer ScienceComputer Science (R0)