Abstract
We characterise an autonomous system as one that has the capability to take decisions independently from human control. This independent and autonomous decision making could give rise to new hazards or hazard causes not present in an equivalent human-controlled system, e.g. through lack of human real-world understanding. Despite the increased adoption of autonomous systems there has been a dearth of research in the area of safety analysis and assurance of decision-making for autonomous systems. This paper is intended to be a first step to fill this gap. We compare and contrast the differing causal models of autonomous and non-autonomous systems, and build on existing safety engineering techniques in order to define a process (Decision Safety Analysis) for the analysis of autonomous decision-making. We show, using a real-world example, how this process supports the development of safety requirements to mitigate hazardous scenarios.
This work is funded by the Assuring Autonomy International Programme https://www.york.ac.uk/assuring-autonomy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Safety and ethics of autonomous systems project overview. Technical report, Royal Academy of Engineering (2020)
Adler, R., Feth, P., Schneider, D.: Safety engineering for autonomous vehicles. In: 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshop (DSN-W), pp. 200–205. IEEE (2016)
Burton, S., Habli, I., Lawton, T., McDermid, J., Morgan, P., Porter, Z.: Mind the gaps: assuring the safety of autonomous systems from an engineering, ethical, and legal perspective. Artif. Intell. 279, 103201 (2020)
International Electrotechnical Commission: IEC 61882 (2016)
Ericson, C.A., et al.: Hazard Analysis Techniques for System Safety. Wiley, Hoboken (2015)
Gleirscher, M., Johnson, N., Karachristou, P., Calinescu, R., Law, J., Clark, J.: Challenges in the safety-security co-assurance of collaborative industrial robots. arXiv preprint arXiv:2007.11099 (2020)
Guiochet, J.: Hazard analysis of human-robot interactions with HAZOP-UML. Saf. Sci. 84, 225–237 (2016)
Harper, C., Caleb-Solly, P.: Towards an ontological framework for environmental survey hazard analysis of autonomous systems. In: SafeAI@ AAAI (2021)
Ishimatsu, T., Leveson, N.G., Thomas, J., Katahira, M., Miyamoto, Y., Nakao, H.: Modeling and hazard analysis using STPA (2010)
Kahneman, D.: Thinking, Fast and Slow. Macmillan (2011)
Klein, G.A., Orasanu, J., Calderwood, R., Zsambok, C.E., et al.: Decision Making in Action: Models and Methods. Ablex Norwood, New Jersey (1993)
Koehler, J.J.: The influence of prior beliefs on scientific judgments of evidence quality. Organ. Behav. Hum. Decis. Process. 56(1), 28–55 (1993)
Koopman, P., Fratrik, F.: How many operational design domains, objects, and events? In: SafeAI@AAAI (2019)
Lesage, B.M.J.R., Alexander, R.: SASSI: safety analysis using simulation-based situation coverage for cobot systems. In: Proceedings of SafeComp 2021, York (2021)
Marcus, G., Davis, E.: Rebooting AI: Building Artificial Intelligence We Can Trust. Vintage (2019)
Medrano-Berumen, C., İlhan Akbaş, M.: Validation of decision-making in artificial intelligence-based autonomous vehicles. J. Inf. Telecommun 5(1), 83–103 (2021)
Oberheid, H., Hasselberg, A., Söffker, D.: Know your options-analysing human decision making in dynamic task environments with state space methods. Hum. Centred Autom. 285–300 (2011)
Osborne, M.: ISA Robot Safety of Decision Making. https://www-users.cs.york.ac.uk/mo705/isarobot.html
Stansbury, R.S., Agah, A.: A robot decision making framework using constraint programming. Artif. Intell. Rev. 38(1), 67–83 (2012)
Walker, G., et al.: Modelling driver decision-making at railway level crossings using the abstraction decomposition space. Cogn. Technol. Work 23(2), 225–237 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Osborne, M., Hawkins, R., McDermid, J. (2022). Analysing the Safety of Decision-Making in Autonomous Systems. In: Trapp, M., Saglietti, F., Spisländer, M., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2022. Lecture Notes in Computer Science, vol 13414. Springer, Cham. https://doi.org/10.1007/978-3-031-14835-4_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-14835-4_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14834-7
Online ISBN: 978-3-031-14835-4
eBook Packages: Computer ScienceComputer Science (R0)