Abstract
Because of its practical advantages, machine learning (ML) is increasingly used for decision-making in numerous sectors. This paper demonstrates that the integral characteristics of ML, such as semi-autonomy, complexity, and non-deterministic modeling have important ethical implications. In particular, these characteristics lead to a lack of insight and lack of comprehensibility, and ultimately to the loss of human control over decision-making. Errors, which are bound to occur in any decision-making process, may lead to great harm and human rights violations. It is important to have a principled way of assigning responsibility for such errors. The integral characteristics of ML, however, pose serious difficulties in defining responsibility and regulating ML decision-making. First, we elaborate on these characteristics and their epistemic and ethical implications. We then analyze possible general strategies for resolving the assignment of moral responsibility and show that, due to the specific way in which ML functions, each potential solution is problematic, whether we assign responsibility to humans, machines, or using hybrid models. Then, we shift focus on an alternative approach that bypasses moral responsibility and attempts to define legal liability independently through solutions such as informed consent and the no-fault compensation system. Both of these solutions prove unsatisfactory because they leave too much room for potential abuses of ML decision-making. We conclude that both ethical and legal solutions are fraught with serious difficulties. These difficulties prompt us to re-weigh the costs and benefits of using ML for high-stake decisions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Non applicable.
Notes
We take artificial neural networks (ANNs) as the paradigmatic type of ML model. Some aspects of our analysis may, however, be relevant for other types of models that give rise to similar issues.
There are different degrees of human involvement in supervised and unsupervised learning, and this implies different degrees of control over the learning process. Supervised learning is characterized by the use of labeled datasets, which requires human intervention to label the data appropriately. In this way humans ‘supervise’ machines to learn how to correctly classify data. In contrast, in unsupervised learning the machine discovers the underlying structures of unlabelled datasets on its own. Admittedly, even unsupervised modeling needs human intervention with regard to validating output variables to be able to learn from data. However, there is still a significant degree of (semi-)autonomy present in ML that is relevant for the epistemic consequences we discuss in 2.2.
The degree of insight into ML models is of course not the same for an engineer who develops these models and for a person who is a complete layman. The engineer, unlike the layman, knows the general principles of functioning of the ML model and in that sense it can be said that for people developing the models they are ‘gray boxes’. However, when we talk about the blackboxness of models, we are referring to the lack of epistemic insight into the aspects of the working of the ML model which applies to experts as well, not just to laymen. Certain aspects of ML models’ functioning are not accessible to any human, and this is what we refer to by ‘blackboxness’ or ‘opacity’.
There have, of course, been many attempts to make the information involved in ML decision-making explainable to humans, by constructing other models trained to produce explanations—the Explainable AI (xAI) Project. We discuss the xAI project and its limitations in Sect. 2.3.
For more details on the differences between unpredictability and other epistemic obstacles such as unexplainability and incomprehensibility, see Yampolskiy (2020).
For example, the legislative request most commonly cited in the xAI literature is the European Union’s General Data Protection Regulation (GDPR), which requests that the subjects of automated decision-making are provided with “meaningful information about the logic involved” in reaching the decision (GDPR, Article 12(2)(f)). The GDPR also states the right of the subjects to “obtain an explanation of the decision reached after such assessment and to challenge the decision” (GDPR, Recital 71). These are the only two mentions of anything related to explanations in this regulation, and these two formulations state two essentially different explanatory requests, one concerning the overall mechanism of the ML model, and the other concerning the path to the model’s reaching a particular decision.
For example, explanatory tools commonly consist in surrogate models that solely attempt to capture the input–output trends of the opaque model they are intended to explain, but they employ entirely different features and are thereby not faithful to the original model’s computations (Rudin 2019).
This understanding of moral agency is what in Moor’s terminology characterizes a ‘full’ moral agent—the only kind of moral agent that we can consider morally responsible. For the complete taxonomy of moral agency, which has become canonical in the literature on this topic, see Moor (2006). We discuss the prerequisites of moral agency in the context of ML in Sect. 4.1.
Certain types of moral transgressions, such as lying in everyday life, are not legally regulated, nor are they expected to be. Giving a false promise to a friend is morally reprehensible, but we will not necessarily end up in court because of it. Of course, some cases of lying such as defamation and false testimony in court are legally regulated, but lying in ordinary daily life usually does not fall into these categories. There is also the possibility that some moral offenses are not legally regulated yet because they have only recently emerged, such as those made possible by the development of technology, but are expected to be regulated in the future.
It may be objected that the internal processes of human decision-making are also inaccessible, and perhaps even more complex than ML decision-making. We cannot look into the heads of others (the so-called ‘problem of other minds’), so we turn to various social procedures developed for inferring the internal states of other humans (see Matthias 2004). These procedures do not make other minds completely transparent, but might provide some kind of insight about the thought processes, intentions and beliefs of others. Similarly, numerous xAI methods are being developed in attempts to gain insight into ML decision-making processes. So why would this make a problem for ML, but not human decision-making? The key difference is that in the case of human decision-making, the locus of responsibility is clear in most cases. In paradigmatic cases, the person who has made a particular decision is the one who is held responsible for it. In the context of ML, however, there are a number of obstacles that make it highly difficult to find the locus of responsibility for the consequences of the decisions. We discuss these obstacles in detail in Sect. 3.2.
This ambiguity of assigning responsibility would not, of course, apply to cases of intentional biasing of data nor to cases of negligent or reckless use, if, for example, a company did not check the ML system for bias, or continued to use it even after bias is discovered.
Having human experts keep track of the correctness of the ML decision-making may seem as a solution to problems of control and responsibility. However, as Matthias points out, “[w]ere it possible to supply every machine with a controlling human expert, nobody would need the machine in the first place” (2004, p. 177). Besides, it does not seem sensible to employ slower or less reliable systems such as humans to keep track of much more efficient and reliable systems and inspect the correctness of the processes. It would defeat the purpose of using ML decision-making in the first place.
There is another direction taken in the literature that focuses on building a moral code into the machines. It is considered that this would prevent unethical machine decisions, as well as diminish harm and human rights violations (Anderson and Anderson 2007; Wallach and Allen 2009). However, this project faces several significant challenges. First, it needs to decide on a particular ethical theory: deontological ethics, virtue ethics, utilitarianism, or some other. Second, the chosen theory must be implementable in the machines, in the sense that it must be translatable into a language that allows computation, and it is still unclear whether this is a feasible task. Finally, even if building a moral code into the machines becomes possible, we still need to decide how to deal with errors if they occur. There is no reason to believe that the ethical machines would be completely infallible. It seems that we would still need a principled way of assigning responsibility and dealing with potential errors. It remains for future research to show how successful the project of building moral machines will be in meeting its challenges. Importantly, the topic of this paper is how to assign responsibility for the decisions of ML systems that are currently in use and that do not have any built-in moral code. The discussion might become different if machines with a built-in moral code are used in the future, depending on how exactly they would function.
We will not enter into the controversy over which of the presented directions is the most adequate from the point of view of moral theory in general. We will only briefly present each of the possible directions and analyze the difficulties they face.
References
Ananny M, Crawford K (2018) Seeing without knowing: limitations of the transparency ideal and its application to algorithmic accountability. New Media Soc 20(3):973–989
Anderson M, Anderson S (2007) Machine ethics: creating an ethical intelligent agent. AI Mag 28(4):15–26
Angwin J, Larson J, Mattu S, Kirchner L (2016) Machine bias: there’s software used across the country to predict future criminals. And it’s biased against blacks. ProPublica. Retrieved November 9, 2021, from https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Apps P (2021) New era of robot war may be underway unnoticed. Reuters. Retrieved September 7, 2021, from https://www.reuters.com/article/apps-drones-idUSL5N2NS2E8
Asaro PM (2012) On banning autonomous weapon systems: human rights, automation, and the dehumanization of lethal decision-making. Int Rev Red Cross 94(886):687–709
Asaro PM (2014) A body to kick, but still no soul to damn: legal perspectives on robotics. In: Lin P, Abney K, Bekey GA (eds) Robot ethics: the ethical and social implications of robotics. MIT Press, pp 169–186
Boge FJ, Grünke P (2019) Computer simulations, machine learning and the Laplacean demon: opacity in the case of high energy physics. In: Kaminski A, Resch M, Gehring P (eds) The science and art of simulation II. Springer
Bryson JJ (2010) Robots should be slaves. In: Wilks Y (ed) Close engagements with artificial companions: key social, psychological, ethical and design issues. John Benjamins, pp 63–74
Burrell J (2016) How the machine ‘thinks’: Understanding opacity in machine learning algorithms. Big Data & Society, 3(1). https://doi.org/10.1177/2053951715622512
Butler D (2016) Tomorrow’s world. Nature 530:399–401
Cornock M (2011) Legal definitions of responsibility, accountability and liability. Nurs Child Young People 23(3):25–26
Fischer JM, Ravizza MSJ (1998) Responsibility and control: a theory of moral responsibility. Cambridge University Press
Flores AW, Lowenkamp CT, Bechtel K (2016) False positives, false negatives, and false analyses: a rejoinder to “Machine bias: there’s software used across the country to predict future criminals. And it’s biased against blacks.” Fed Probat J 80(2):38–46
Floridi L, Cowls J, Beltrametti M, Chatila R, Chazerand P, Dignum V, Luetge C, Madelin R, Pagallo U, Rossi F, Schafer B, Valcke P, Vayena E (2018) AI4People—an ethical framework for a good AI society: opportunities, risks, principles, and recommendations. Mind Mach 28:689–707
Gaine WJ (2003) No-fault compensation systems. BMJ 326(7397):997–998
Gilpin LH, Bau D, Yuan BZ, Bajwa A, Specter M, Kagal L (2018) Explaining explanations: an overview of interpretability of machine learning. In: Proceedings of the 2018 IEEE 5th international conference on Data Science and Advanced Analytics (DSAA). IEEE, pp 80–89
Goertzel B (2002) Thoughts on AI morality. Dyn Psychol Int Interdiscip J Complex Ment Process. Retrieved October 31, 2021, from http://www.goertzel.org/dynapsyc/2002/AIMorality.htm
Goh YC, Cai XQ, Theseira W, Ko G, Khor KA (2020) Evaluating human versus machine learning performance in classifying research abstracts. Scientometrics 125:1197–1212
Goodman B, Flaxman S (2017) EU regulations on algorithmic decision-making and a ‘Right to Explanation.’ AI Mag 38(3):50–57
Grossmann J, Wiesbrock HW, Motta M (2021) Testing ML-based systems. Federal Ministry for Economic Affairs and Energy. https://docbox.etsi.org/mts/mts/05-CONTRIBUTIONS/2022/MTS(22)086017_Testing_ML-based_Systems.pdf
Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D (2018) A survey of methods for explaining black box models. ACM Comput Surv 51(5):1–42
Gunkel DJ (2020) Mind the gap: responsible robotics and the problem of responsibility. Ethics Inf Technol 22:307–320
Hall JS (2001) Ethics for machines. Kurzweil Essays. Retrieved June 15, 2021, from KurzweilAI.net http://www.kurzweilai.net/ethics-for-machines
Hanson FA (2009) Beyond the skin bag: on the moral responsibility of extended agencies. Ethics Inf Technol 11:91–99
Hart E (2019) Machine learning 101: the what, why, and how of weighting. KDnuggets. Retrieved May 21, 2021, from https://www.kdnuggets.com/2019/11/machine-learning-what-why-how-weighting.html
Henry LM, Larkin ME, Pike ER (2015) Just compensation: a no-fault proposal for research-related injuries. J Law Biosci 2(3):645–668
Hoffman RR, Mueller ST, Klein G, Litman J (2018) Metrics for explainable AI: challenges and prospects. XAI Metrics. Retrieved October 1, 2021, from https://arxiv.org/ftp/arxiv/papers/1812/1812.04608.pdf
Humphreys P (2004) Extending ourselves: computational science, empiricism, and scientific method. Oxford University Press
Humphreys P (2009) The philosophical novelty of computer simulation methods. Synthese 169:615–626
Johnson DG (2006) Computer systems: moral entities but not moral agents. Ethics Inf Technol 8(4):195–204
Johnson DG, Miller KW (2008) Un-making artificial moral agents. Ethics Inf Technol 10(2–3):123–133
Lauret J (2019) Amazon’s sexist AI recruiting tool: how did it go so wrong? Medium. Retrieved November 9, 2021, from https://becominghuman.ai/amazons-sexist-ai-recruiting-tool-how-did-it-go-so-wrong-e3d14816d98e
Lee J (2020) Is artificial intelligence better than human clinicians in predicting patient outcomes? J Med Internet Res 22(8):e19918. https://doi.org/10.2196/19918
Lipton ZC (2016) The mythos of model interpretability. In: 2016 ICML workshop on human interpretability in machine learning (WHI 2016). New York. https://arxiv.org/abs/1606.03490
Lunney M, Oliphant K (2013) Tort law, 5th edn. Oxford University Press
Matthias A (2004) The responsibility gap: ascribing responsibility for the actions of learning automata. Ethics Inf Technol 6(3):175–183
McKenna M (2008) Putting the lie on the control condition for moral responsibility. Philos Stud 139:29–37
Mehta S (2022) Deterministic vs stochastic machine learning [Blog post]. https://analyticsindiamag.com/deterministic-vs-stochastic-machine-learning/
Miller T (2017) Explanation in artificial intelligence: insights from the social science. Artif Intell 267:1–38
Mittelstadt B (2019) Principles alone cannot guarantee ethical AI. Nat Mach Intell 1:501–507
Mittelstadt B, Russell C, Wachter S (2019) Explaining explanations in AI. In: FAT* ’19: conference on fairness, accountability, and transparency (FAT* ’19). Retrieved October 30, 2021, from https://arxiv.org/pdf/1811.01439.pdf
Molnar C (2019) Interpretable Machine Learning. Available online: https://christophm.github.io/interpretable-mlbook/
Moor J (2006) The nature, importance and difficulty of machine ethics. IEEE Intelligent Systems 21(4): 18-21
Mowshowitz A (2008) Technology as excuse for questionable ethics. AI Soc 22(3):271–282
Nissenbaum H (1996) Accountability in a computerized society. Sci Eng Ethics 2(1):25–42
Ombach J (2014) A short introduction to stochastic optimization. Schedae Informaticae 23:9–20
Paez A (2019) The pragmatic turn in explainable artificial intelligence (XAI). Mind Mach 29:441–459
Pant K (2021) AI in the courts [Blog post]. Retrieved from https://indianexpress.com/article/opinion/artificial-intelligence-in-the-courts-7399436/
Price M (2019) Hospital ‘risk scores' prioritize white patients. Science. Retrieved November 9, 2021, from https://www.science.org/content/article/hospital-risk-scores-prioritize-white-patients
Ribera TM, Lapedriza A (2019) Can we do better explanations? A proposal of user-centered explainable AI. In joint Proceedings of the ACM IUI 2019 workshops
Rudin C (2019) stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1(5):206–215
Russ M (2021) Artificial intelligence, machine learning, and deep learning—what is the difference and why it matters [Blog post]. Retrieved from https://bluehealthintelligence.com/how-to-know-the-difference-between-artificial-intelligence-machine-learning-and-deep-learning-and-why-it-matters/
Russell SJ, Norvig P (eds) (2016) Artificial intelligence: a modern approach. Pearson Education Limited, Cham
Samek W, Montavon G, Vedaldi A, Hansen LK, Müller KR (eds) (2019) Explainable AI: interpreting, explaining and visualizing deep learning. Springer
Schembera B (2017) Myths of Simulation. In: Resch MM, Kaminski A, Gehring P (eds) The science and art of simulation I: exploring—understanding—knowing. Springer, Cham, pp 51–63
Sidelov P (2021) Machine learning in banking: top use cases [Blog post]. Retrieved from https://sdk.finance/top-machine-learning-use-cases-in-banking/
Siponen M (2004) A pragmatic evaluation of the theory of information ethics. Ethics Inf Technol 6(4):279–290
Sparrow R (2007) Killer robots. J Appl Philos 24(1):62
Srećković S, Berber A, Filipović N (2022) The automated Laplacean demon: how ML challenges our views on prediction and explanation. Mind Mach. https://doi.org/10.1007/s11023-021-09575-6
Sullins JP (2006) When is a robot a moral agent? Int Rev Inf Ethics 6(12):23–30
Talbert M (2022) Moral responsibility. In: Zalta EN, Nodelman U (eds) The Stanford encyclopedia of philosophy (Fall 2022 edition). https://plato.stanford.edu/archives/fall2022/entries/moral-responsibility/
Tkachenko N (2021) Machine learning in healthcare: 12 real-world use cases to know [Blog post]. Retrieved from https://nix-united.com/blog/machine-learning-in-healthcare-12-real-world-use-cases-to-know/#:~:text=One%20of%20the%20uses%20of,decision%2Dmaking%20and%20patient%20care.
Turing A (1999) Computing machinery and intelligence. In: Meyer PA (ed) Computer media and communication: a reader. Oxford University Press, pp 37–58
UNI Global Union (2018) 10 principles for ethical AI. UNI Global Union, February 21, 2021. http://www.thefutureworldofwork.org/opinions/10-principles-for-ethical-ai/
Varshney KR, Alemzadeh H (2017) On the safety of machine learning: cyber-physical systems, decision sciences, and data products. Big Data 5(3):246–255
Verbeek PP (2011) Moralizing technology: understanding and designing the morality of things. University of Chicago Press
Wachter S, Mittelstadt B, Floridi L (2016) Why a right to explanation of automated decision-making does not exist in the general data protection regulation. Int Data Privacy Law 7(2):76–99
Wachter S, Mittelstadt B, Russell C (2018) Counterfactual explanations without opening the black box: automated decisions and the GDPR. Harv J Law Technol 31(2):841–887
Wallach W, Allen C (2009) Moral machines: teaching robots right from wrong. Oxford University Press
Wang F, Rudin C, McCormick TH, Gore JL (2019) Modeling recovery curves with application to prostatectomy. Biostatistics 20(4):549–564
Wang H, Shuai P, Deng Y et al (2022) A correlation-based feature analysis of physical examination indicators can help predict the overall underlying health status using machine learning. Sci Rep 12:19626
Wexler R (2017) When a computer program keeps you in jail: how computers are harming criminal justice. New York Times. Retrieved October 3, 2021, https://www.nytimes.com/2017/06/13/opinion/how-computers-are-harming-criminal-justice.html
Wyber R, Vaillancourt S, Perry W, Mannava P, Folaranmi T, Celi LA (2015) Big data in global health: improving health in low- and middle-income countries. Bull World Health Organ 93(3):203–208
Yampolskiy R (2020) Unexplainability and incomprehensibility of AI. J Artif Intell Conscious 7(2):277–291
Yeung K (2019) Responsibility and AI: a study of the implications of advanced digital technologies (including AI systems) for the concept of responsibility within a human rights framework. Council of Europe Study Series. Council of Europe
Zednik C (2019) Solving the black box problem: a normative framework for explainable artificial intelligence. Philos Technol 34:265–288
Zerilli J, Knott A, Maclaurin J, Gavaghan C (2019) Transparency in algorithmic and human decision-making: is there a double standard? Philos Technol 32:661–683
Zhao T, Dai E, Shu K, Wang S (2022) Towards fair classifiers without sensitive attributes: exploring biases in related features. In: Conference: WSDM '22: the fifteenth ACM international conference on web search and data mining, pp 1433–1442. https://doi.org/10.1145/3488560.3498493
Zimmerman MJ (1997) Moral responsibility and ignorance. Ethics 107(3):410–426
Acknowledgements
We would like to thank Nenad Filipović for the engaged discussion and helpful comments on the early versions of this paper.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Berber, A., Srećković, S. When something goes wrong: Who is responsible for errors in ML decision-making?. AI & Soc (2023). https://doi.org/10.1007/s00146-023-01640-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00146-023-01640-1