This paper examines the ethical pitfalls and challenges that non-ethicists, such as researchers and programmers in the fields of computer science, artificial intelligence and robotics, face when building moral machines. Whether ethics is “computable” depends on how programmers understand ethics in the first place and on the adequacy of their understanding of the ethical problems and methodological challenges in these fields. Researchers and programmers face at least two types of problems due to their general lack of ethical knowledge or expertise. The first type is so-called rookie mistakes, which could be addressed by providing these people with the necessary ethical knowledge. The second, more difficult methodological issue concerns areas of peer disagreement in ethics, where no easy solutions are currently available. This paper examines several existing approaches to highlight the ethical pitfalls and challenges involved. Familiarity with these and similar problems can help programmers to avoid pitfalls and build better moral machines. The paper concludes that ethical decisions regarding moral robots should be based on avoiding what is immoral (i.e. prohibiting certain immoral actions) in combination with a pluralistic ethical method of solving moral problems, rather than relying on a particular ethical approach, so as to avoid a normative bias.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
In this paper, the concept of computability is used in a rather broad sense. It concerns the way in which a machine is built to make ethical decisions that generally seek to resemble the moral reasoning and decision making of human beings.
For example, Anderson and Anderson (2007: 15) observe, “Using Moor’s terminology, most of those working on machine ethics would say that the ultimate goal is to create a machine that is an explicit ethical agent.” This might be true, but we should be prepared for the advent of IRs who are full ethical agents, regardless of how long it may take for such IRs to become a reality.
See Allen et al. (2000: 251–261) for the interesting idea of the so-called Moral Turing Test (MTT) that robots would pass if they (1) give correct answers to moral dilemmas and (2) offer the appropriate justification for their claims. This would amount to understanding ethics.
It is not claimed that the approaches analysed in the table are the only important ones, but they are a representative sample of the existing range of accounts that have been extensively discussed in recent years.
The model presented by Wallach et al. is not necessarily inadequate with respect to how moral decision making works in an empirical sense, but their approach is descriptive rather than normative by nature. Therefore, their empirical model does not solve the normative problem of how moral machines should actually act. Descriptive ethics and normative ethics are two different things. The first tells us how human beings make moral decisions; the latter is concerned with how we should act.
Hall (2011: 523) claims, “In the long run, AIs will run everything, and should, because they can do a better job than humans. Not only will their intellectual prowess exceed ours, but their moral judgment as well.”
Dietrich argues that human beings, as biological entities, might have a genetic predisposition for selfish behaviour as a survival mechanism with respect to competition with others. Machines lack this predisposition and could be made in such a way that they both possess the good features of human beings and follow ethical principles. They could then be seen as “Homo sapiens 2.0”—enhanced or improved human beings.
One might also claim here that the ability of moral machines to perform “better” than humans in the realm of ethics relates to the fact that they do not suffer from moral imperfections—e.g. prejudice, selfishness, partiality, weakness of will, and being carried away by emotions—in their ethical reasoning and decision making.
See Pereira and Saptawijaya (2011) for a detailed description of how their system, ACORDA, uses prospective logic programming to solve various dilemmas like the trolley case.
See, for example, the interesting programmable “moral machine” developed by Scalable Cooperation at MIT Media Lab concerning the question “Between whom is the self-driving car deciding?” in a case where the car’s brakes fail. It is a “platform for gathering a human perspective on moral decisions made by machine intelligence, such as self-driving cars … where a driverless car must choose the lesser of two evils, such as killing two passengers or five pedestrians. As an outside observer, you judge which outcome you think is more acceptable” (www.moralmachine.mit.edu, accessed 2 July 2018). However, the MIT trial does not involve cars working in concert, but only in isolation or in a way that involves human–robot interaction. Self-driving cars may eventually be part of a larger system in which machines will communicate with each other and thereby solve upcoming problems more fully than can occur when each machine functions in isolation, particularly in the context of automated driving (Borenstein et al. 2017).
The idea that machines might come up with new or unexpressed ethical principles has been advanced by Anderson and Anderson (2011b: 476–492, esp. 479).
Abney, K. (2014). Robotics, ethical theory, and metaethics: A guide for the perplexed. In P. Lin, K. Abney, & G. Bekey (Eds.), Robot ethics: The ethical and social implications of robotics (pp. 35–52). Cambridge, MA: MIT Press.
Aletras, N., Tsarapatsanis, D., Preoţiuc-Pietro, D., & Lampos, V. (2016). Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective. PeerJ Computer Science,2(e93), 1–19.
Allen, C., Varner, G., & Zinser, J. (2000). Prolegomena to any future artificial moral agent. Journal of Experimental & Theoretical Artificial Intelligence,12(3), 251–261.
Allen, C., Wallach, W., & Smit, I. (2011). Why machine ethics? In M. Anderson & S. Anderson (Eds.), Machine ethics (pp. 51–61). Cambridge, MA: Cambridge University Press.
Anderson, S. L. (2011a). How machines might help us achieve breakthroughs in ethical theory and inspire us to behave better. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 524–5309). Cambridge, MA: Cambridge University Press.
Anderson, S. L. (2011b). Philosophical concerns with machine ethics. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 162–167). Cambridge, MA: Cambridge University Press.
Anderson, S. L. (2011c). Machine metaethics. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 21–27). Cambridge, MA: Cambridge University Press.
Anderson, M., & Anderson, S. L. (2007). Machine ethics: Creating an ethical intelligent agent. AI Magazine,28(4), 15–26.
Anderson, M., & Anderson, S. L. (2011a). Machine ethics. Cambridge, MA: Cambridge University Press.
Anderson, M., & Anderson, S. L. (2011b). A prima facie duty approach to machine ethics. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 476–492). Cambridge, MA: Cambridge University Press.
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. Accessed 8 March 2018.
Bayertz, K. (1991). Praktische Philosophie als angewandte Ethik”. In K. Bayertz (Ed.), Praktische Philosophie: Grundorientierungen angewandter Ethik (pp. 7–47). Hamburg: Rowohlt Taschenbuch Verlag.
Bekey, G. A. (2014). Current trends in robotics: Technology and ethics. In P. Lin, K. Abney, & G. Bekey (Eds.), Robot ethics: The ethical and social implications of robotics (pp. 17–34). Cambridge, MA: MIT Press.
Borenstein, J., Herkert, J. R., & Miller, K. W. (2017). Self-driving cars and engineering ethics: The need for a system level analysis. Science and Engineering Ethics. https://doi.org/10.1007/s11948-017-0006-0.
Bringsjord, S., & Taylor, J. (2014). The divine-command approach to robot ethics. In P. Lin, K. Abney, & G. Bekey (Eds.), Robot ethics: The ethical and social implications of robotics (pp. 85–108). Cambridge, MA: MIT Press.
Brody, B. (1988). Life and death decision making. Oxford: Oxford University Press.
Clouser, K. D., & Gert, B. (1990). A critique of principlism. Journal of Medicine and Philosophy,15(2), 219–236.
Coeckelbergh, M. (2014). The moral standing of machines: Towards a relational and non-Cartesian moral hermeneutics. Philosophy & Technology,27(1), 61–77.
Copeland, J. (1993). Artificial intelligence: A philosophical introduction. Oxford: Wiley.
Dehghani, M., Forbus, K., Tomai, E., & Klenk, M. (2011). An integrated reasoning approach to moral decision making. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 422–441). Cambridge, MA: Cambridge University Press.
Dietrich, E. (2011). Homo sapiens 2.0: Building the better robots of our nature. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 531–538). Cambridge, MA: Cambridge University Press.
Floridi, L. (2011). On the morality of artificial agents. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 184–212). Cambridge: Cambridge University Press.
Floridi, L., & Sanders, J. W. (2004). On the morality of artificial agents. Minds and Machines,14(3), 349–379.
Foot, P. (1967). The problem of abortion and the doctrine of double effect. Oxford Review,5, 5–15.
Gips, J. (2011). Towards the ethical robot. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 244–253). Cambridge, MA: Cambridge University Press.
Giubilini, A., & Savulescu, J. (2018). The artificial moral advisor. The “Ideal Observer” meets artificial intelligence. Philosophy & Technology,31(2), 169–188.
Gordon, J.-S. (2011). Global ethics and principlism. Kennedy Institute of Ethics Journal,21(3), 251–276.
Gordon, J.-S. (2015). Human rights and cultural identity. Baltic Journal of Law & Politics,8(2), 112–135.
Gordon, J. S. (2016a). Ethics as a method. In J. Schildmann, J. S. Gordon, & J. Vollmann (Eds.), Clinical ethics consultation: Theories and methods, implementation, evaluation (pp. 11–20). London: Routledge.
Gordon, J.-S. (2016b). Should moral enhancement be compulsory? Ethical Perspectives,23(2), 277–305.
Gordon, J.-S. (2018). What do we owe to intelligent robots? AI & Society. https://doi.org/10.1007/s00146-018-0844-6.
Gordon, J.-S., Rauprich, O., & Vollmann, J. (2011). Applying the four-principle approach. Bioethics,25(6), 293–300.
Guarini, M. (2006). Particularism and the classification and reclassification of moral cases. IEEE Intelligent Systems,21(4), 22–28.
Gunkel, D. J., & Bryson, J. (2014). The machine as moral agent and patient. Philosophy & Technology,27(1), 5–142.
Hall, J. S. (2011). Ethics for self-improving machines. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 512–523). Cambridge, MA: Cambridge University Press.
Hughes, J. (2014). Compassionate AI and selfless robots: A Buddhist approach. In P. Lin, K. Abney, & G. Bekey (Eds.), Robot ethics: The ethical and social implications of robotics (pp. 69–84). Cambridge, MA: MIT Press.
Johnson, A. M., & Axinn, S. (2014). Acting versus being moral: The limits of technological moral actors. In Proceedings of the IEEE 2014 international symposium on ethics in engineering, science, and technology (pp. 30(1)–30(4)). Piscataway, NJ: IEEE Press.
Kant, I., & Paton, H. J. (2009). Groundwork of the metaphysic of morals. New York: Harper Perennial Modern Classics.
Katz, D. M., Bommarito, M. J., II, & Blackman, J. (2017). A general approach for predicting the behavior of the Supreme Court of the United States. PLoS ONE,12(4), 1–18.
Lin, P., Abney, K., & Bekey, G. A. (Eds.). (2014). Robot ethics: The ethical and social implications of robotics. Intelligent robotics and autonomous agents. Cambridge, MA: MIT Press.
Macklin, R. (1999). Against relativism. Cultural diversity and the search for ethical universals in medicine. New York: Oxford University Press.
McLaren, B. M. (2011). Computational models of ethical reasoning: Challenges, initial steps, and future directions. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 297–315). Cambridge, MA: Cambridge University Press.
Moor, J. H. (2006). The nature, importance, and difficulty of machine ethics. Research Gate,21(4), 18–21.
Nadeau, J. E. (2006). Only androids can be ethical. In K. Ford & C. Glymour (Eds.), Thinking about android epistemology (pp. 241–248). Cambridge, MA: MIT Press.
Pereira, L. M., & Saptawijaya, A. (2011). Modeling morality with prospective logic. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 398–421). Cambridge, MA: Cambridge University Press.
Picard, R. (1997). Affective computing. Cambridge: The MIT Press.
Pontier, M. A., & Hoorn, J. F. (2012). Toward machines that behave ethically better than humans do. In N. Miyake, B. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th international annual conference of the cognitive science society (pp. 2198–2203). CogSci’12.
Powers, T. (2011). Prospectives for a Kantian machine. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 464–475). Cambridge, MA: Cambridge University Press.
Rodogno, R. (2016). Robots and the limits of morality. In M. Norskov (Ed.), Social robots: Boundaries, potential, challenges. London: Routledge. http://pure.au.dk/portal/files/90856828/Robots_and_the_Limits_of_Morality.pdf. Accessed 12 March 2016.
Ross, W. D. (1930). The right and the good. Oxford: Clarendon Press.
Rzepka, R., & Araki, K. (2005). What statistics could do for ethics? The idea of common sense processing based safety valve. In AAAI fall symposium on machine ethics, Technical Report FS-05-06: 85-87.
Sullins, J. P. (2011). When is a robot a moral agent? In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 151–161). Cambridge, MA: Cambridge University Press.
Thomson, J. J. (1976). Killing, letting die, and the trolley problem. The Monist,59(2), 204–217.
Torrance, S. (2008). Ethics and consciousness in artificial agents. AI & Society,22(4), 495–521.
Wallach, W., & Allen, C. (2010). Moral machines: Teaching robots right from wrong. Oxford: Oxford University Press.
Wallach, W., Franklin, S., & Allen, C. (2010). A conceptual and computational model of moral decision making in human and artificial agents. Topics in Cognitive Science,2(3), 454–485.
I would like to thank the anonymous reviewers for their valuable comments. This research is funded by the European Social Fund according to the activity ‘Improvement of researchers’ qualification by implementing world-class R&D projects of Measure No. 09.3.3-LMT-K-712.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Gordon, J. Building Moral Robots: Ethical Pitfalls and Challenges. Sci Eng Ethics 26, 141–157 (2020). https://doi.org/10.1007/s11948-019-00084-5
- Moral machines
- Full ethical agents
- Ethical expertise
- Programming ethics
- Moral pluralism