Abstract
Robot ethics encompasses ethical questions about how humans should design, deploy, and treat robots; machine morality encompasses questions about what moral capacities a robot should have and how these capacities could be computationally implemented. Publications on both of these topics have doubled twice in the past 10 years but have often remained separate from one another. In an attempt to better integrate the two, I offer a framework for what a morally competent robot would look like (normally considered machine morality) and discuss a number of ethical questions about the design, use, and treatment of such moral robots in society (normally considered robot ethics). Instead of searching for a fixed set of criteria of a robot’s moral competence I identify the multiple elements that make up human moral competence and probe the possibility of designing robots that have one or more of these human elements, which include: moral vocabulary; a system of norms; moral cognition and affect; moral decision making and action; moral communication. Juxtaposing empirical research, philosophical debates, and computational challenges, this article adopts an optimistic perspective: if robotic design truly commits to building morally competent robots, then those robots could be trustworthy and productive partners, caretakers, educators, and members of the human community. Moral competence does not resolve all ethical concerns over robots in society, but it may be a prerequisite to resolve at least some of them.
Similar content being viewed by others
Notes
Numbers are derived from an EBSCO database search using “robot* and ethic*” as subject search terms and restricting to contributions in journals and books.
This volume appeared as part of MIT Press’s series on “Intelligent robotics and autonomous agents.” Notably, ethics was addressed as the 19th topic in the series, 15 years after the series commenced.
Strictly speaking, this requirement holds only for truthful explanations, which I hope will be the default for social robots.
References
Alicke, M. D. (2000). Culpable control and the psychology of blame. Psychological Bulletin, 126, 556–574. doi:10.1037//0033-2909.126.4.556.
Allen, C. (2011). The future of moral machines. The New York Times: Opinionator. Retrieved December 29, 2014, from http://opinionator.blogs.nytimes.com/2011/12/25/the-future-of-moral-machines/.
Anderson, M., & Anderson, S. (2011). Machine ethics. Cambridge: Cambridge University Press.
Antaki, C. (1994). Explaining and arguing: The social organization of accounts. London: Sage.
Arkin, R. C. (2009). Governing lethal behavior in autonomous robots. Boca Raton, FL: CRC Press.
Asaro, P. M. (2006). What should we want from a robot ethic? International Review of Information Ethics, 6, 9–16.
Avramova, Y. R., & Inbar, Y. (2013). Emotion and moral judgment. Wiley Interdisciplinary Reviews Cognitive Science, 4, 169–178. doi:10.1002/wcs.1216.
Baumeister, R. F., & Leary, M. R. (1995). The need to belong: Desire for interpersonal attachments as a fundamental human motivation. Psychological Bulletin, 117, 497–529. doi:10.1037/0033-2909.117.3.497.
Bello, P. (2012). Cognitive foundations for a computational theory of mindreading. Advances in Cognitive Systems, 1, 59–72.
Bicchieri, C. (2006). The grammar of society: The nature and dynamics of social norms. New York, NY: Cambridge University Press.
Blomkamp, N., Kinberg, S. (Producers), & Blomkamp, N. (Director). (2015). Chappie [Motion picture]. USA: Sony Pictures Home Entertainment.
Brachman, R. J. (2002). Systems that know what they’re doing. IEEE Intelligent Systems, 17, 67–71. doi:10.1109/MIS.2002.1134363.
Breazeal, C. L. (2002). Designing sociable robots. Cambridge, MA: MIT Press.
Bringsjord, S. (2009). But perhaps robots are essentially non-persons. Erwägen Wissen Ethik, 20, 193–195.
Bringsjord, S., Arkoudas, K., & Bello, P. (2006). Toward a general logicist methodology for engineering ethically correct robots. Intelligent Systems, IEEE, 21, 38–44.
Calverley, D. J. (2006). Android science and animal rights, does an analogy exist? Connection Science, 18, 403–417. doi:10.1080/09540090600879711.
Coates, D. J., & Tognazzini, N. A. (2012). The contours of blame. In D. J. Coates & N. A. Tognazzini (Eds.), Blame: Its nature and norms (pp. 3–26). New York, NY: Oxford University Press.
Coeckelbergh, M. (2010). Robot rights? Towards a social-relational justification of moral consideration. Ethics and Information Technology, 12, 209–221. doi:10.1007/s10676-010-9235-5.
Cox, M. T. (2011). Metareasoning, monitoring, and self-explanation. In M. T. Cox & A. Raja (Eds.), Metareasoning (pp. 131–149). Cambridge, MA: The MIT Press.
Csibra, G., & Gergely, G. (2009). Natural pedagogy. Trends in Cognitive Sciences, 13, 148–153. doi:10.1016/j.tics.2009.01.005.
Cushman, F. (2008). Crime and punishment: Distinguishing the roles of causal and intentional analyses in moral judgment. Cognition, 108, 353–380. doi:10.1016/j.cognition.2008.03.006.
Cushman, F., & Young, L. (2011). Patterns of moral judgment derive from nonmoral psychological representations. Cognitive Science, 35, 1052–1075. doi:10.1111/j.1551-6709.2010.01167.x.
DeBaets, A. M. (2014). Can a robot pursue the good? Exploring artificial moral agency. Journal of Evolution and Technology, 24, 76–86.
Dersley, I., & Wootton, A. (2000). Complaint sequences within antagonistic argument. Research on Language and Social Interaction, 33, 375–406. doi:10.1207/S15327973RLSI3304_02.
Eisenberg, N. (2000). Emotion, regulation, and moral development. Annual Review of Psychology, 51, 665–697.
Emde, R. N. (1992). Social referencing research: Uncertainty, self, and the search for meaning. In S. Feinman (Ed.), Social referencing and the social construction of reality in infancy (pp. 79–94). New York, NY: Plenum Press.
Fehr, E., & Fischbacher, U. (2004). Third-party punishment and social norms. Evolution and Human Behavior, 25, 63–87. doi:10.1016/S1090-5138(04)00005-4.
Fisher, M., Spielberg, S., & Weaver, B. (2014). Extant [Television series]. Los Angeles: CBS.
Fiske, S. T., & Taylor, S. E. (2008). Social cognition: From brains to culture (1st ed.). Boston, MA: McGraw-Hill.
Flack, J. C., & de Waal, F. B. M. (2000). “Any animal whatever”. Darwinian building blocks of morality in monkeys and apes. Journal of Consciousness Studies, 7, 1–29.
Floridi, L., & Sanders, J. W. (2004). On the morality of artificial agents. Minds and Machines, 14, 349–379. doi:10.1023/B:MIND.0000035461.63578.9d.
Ford, K. M., & Hayes, P. J. (1991). Reasoning agents in a dynamic world: The frame problem. Greenwich, CT: JAI Press.
Fridin, M. (2014). Kindergarten social assistive robot: First meeting and ethical issues. Computers in Human Behavior, 30, 262–272. doi:10.1016/j.chb.2013.09.005.
Garcia, E., Jimenez, M. A., De Santos, P. G., & Armada, M. (2007). The evolution of robotics research. IEEE Robotics Automation Magazine, 14, 90–103. doi:10.1109/MRA.2007.339608.
Gilovich, T., Keltner, D., & Nisbett, R. E. (2013). Social psychology (3rd ed.). New, NY: W.W. Norton & Co.
Grau, C. (2011). There is no “I” in “Robot”: Robots and utilitarianism. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 451–463). Cambridge: Cambridge University Press.
Gray, K., Young, L., & Waytz, A. (2012). Mind perception is the essence of morality. Psychological Inquiry, 23, 101–124. doi:10.1080/1047840X.2012.651387.
Greene, J. D., Nystrom, L. E., Engell, A. D., Darley, J. M., & Cohen, J. D. (2004). The neural bases of cognitive conflict and control in moral judgment. Neuron, 44, 389–400. doi:10.1016/j.neuron.2004.09.027.
Guglielmo, S., Monroe, A. E., & Malle, B. F. (2009). At the heart of morality lies folk psychology. Inquiry: An Interdisciplinary Journal of Philosophy, 52, 449–466. doi:10.1080/00201740903302600.
Gunkel, D. J. (2014). A vindication of the rights of machines. Philosophy & Technology, 27, 113–132. doi:10.1007/s13347-013-0121-z.
Haidt, J. (2001). The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108, 814–834. doi:10.1037/0033-295X.108.4.814.
Hamlin, J. K. (2013). Moral judgment and action in preverbal infants and toddlers: Evidence for an innate moral core. Current Directions in Psychological Science, 22, 186–193. doi:10.1177/0963721412470687.
Harenski, C. L., Harenski, K. A., Shane, M. S., & Kiehl, K. A. (2010). Aberrant neural processing of moral violations in criminal psychopaths. Journal of Abnormal Psychology, 119, 863–874.
Heath, J. (2001). Communicative action and rational choice. Studies in contemporary German social thought. Cambridge, MA: MIT Press.
Hilton, D. J. (2007). Causal explanation: From social perception to knowledge-based causal attribution. In A. W. Kruglanski & E. T. Higgins (Eds.), Social psychology: Handbook of basic principles (2nd ed., pp. 232–253). New York, NY: Guilford Press.
Hoffman, M. L. (2008). Empathy and prosocial behavior. In M. Lewis, J. M. Haviland-Jones, & L. F. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 440–455). New York, NY: Guilford Press.
Hofmann, B. (2013). Ethical challenges with welfare technology: A review of the literature. Science and Engineering Ethics, 19, 389–406.
Huebner, B., Dwyer, S., & Hauser, M. (2009). The role of emotion in moral psychology. Trends in Cognitive Sciences, 13, 1–6. doi:10.1016/j.tics.2008.09.006.
Hutcherson, C. A., & Gross, J. J. (2011). The moral emotions: A social—functionalist account of anger, disgust, and contempt. Journal of Personality and Social Psychology, 100, 719–737. doi:10.1037/a0022408.
Johnson, A. M., & Axinn, S. (2013). The morality of autonomous robots. Journal of Military Ethics, 12, 129–141. doi:10.1080/15027570.2013.818399.
Kahn, Jr., P. H., Kanda, T., Ishiguro, H., Gill, B. T., Ruckert, J. H., Shen, S., Gary, H. E., et al. (2012). Do people hold a humanoid robot morally accountable for the harm it causes?. Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction (pp. 33–40). New York, NY: ACM. doi:10.1145/2157689.2157696.
Kibble, R. (2012). Can an unmanned drone be a moral agent? Ethics and accountability in military robotics. In D. J. Gunkel, J. J. Bryson, & S. Torrance (Eds.), The machine question: AI, ethics and moral responsibility (Proceedings of symposium “Machine Question: AI, Ethics, and Moral Responsibility” AISB/IACAP 2012) (pp. 62–67). The Society for the Study of Artificial Intelligence and Simulation of Behaviour.
Knobe, J. (2010). Person as scientist, person as moralist. Behavioral and Brain Sciences, 33, 315–329. doi:10.1017/S0140525X10000907.
Knobe, J., & Fraser, B. (2008). Causal judgment and moral judgment: Two experiments. Moral psychology (Vol. 2): The cognitive science of morality: Intuition and diversity (Vol. 2, pp. 441–447). Cambridge, MA: MIT Press.
Kohlberg, L. (1984). The psychology of moral development: The nature and validity of moral stages. San Francisco, CA: Harper & Row.
Lin, P. (2013). The ethics of autonomous cars. The Atlantic. Retrieved September 30, 2014, from http://www.theatlantic.com/technology/archive/2013/10/the-ethics-of-autonomous-cars/280360/.
Lin, P., Abney, K., & Bekey, G. A. (Eds.). (2012). Robot ethics: The ethical and social implications of robotics. Cambridge, MA: MIT Press.
Littman, M. L. (2001). Value-function reinforcement learning in Markov games. Cognitive Systems Research, 2, 55–66. doi:10.1016/S1389-0417(01)00015-8.
Lomas, M., Chevalier, R., Cross, E. V., Garrett, R. C., Hoare, J., & Kopack, M. (2012). Explaining robot actions. Proceedings of the 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (pp. 187–188). Boston, MA.
Luo, Q., Nakic, M., Wheatley, T., Richell, R., Martin, A., & Blair, R. J. R. (2006). The neural basis of implicit moral attitude—An IAT study using event-related fMRI. NeuroImage, 30, 1449–1457. doi:10.1016/j.neuroimage.2005.11.005.
Malle, B. F. (1999). How people explain behavior: A new theoretical framework. Personality and Social Psychology Review, 3, 23–48. doi:10.1207/s15327957pspr0301_2.
Malle, B. F. (2004). How the mind explains behavior: Folk explanations, meaning, and social interaction. Cambridge, MA: MIT Press.
Malle, B. F. (2011). Time to give up the dogmas of attribution: A new theory of behavior explanation. In M. P. Zanna & J. M. Olson (Eds.), Advances of experimental social psychology (Vol. 44, pp. 297–352). San Diego, CA: Academic Press.
Malle, B. F., & Dickert, S. (2007). Values. In R. F. Baumeister & K. D. Vohs (Eds.), The encyclopedia of social psychology. Thousand Oaks, CA: Sage.
Malle, B. F., Guglielmo, S., & Monroe, A. E. (2014). A theory of blame. Psychological Inquiry, 25, 147–186. doi:10.1080/1047840X.2014.877340.
Malle, B. F., & Scheutz, M. (2014). Moral competence in social robots. IEEE International Symposium on Ethics in Engineering, Science, and Technology (pp. 30–35). Presented at the IEEE International Symposium on Ethics in Engineering, Science, and Technology, June, Chicago, IL: IEEE.
Malle, B. F., Scheutz, M., Arnold, T., Voiklis, J., & Cusimano, C. (2015). Sacrifice one for the good of many? People apply different moral norms to human and robot agents. HRI’15: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (pp. 117–124). New York, NY: ACM.
McCullough, M. E., Kurzban, R., & Tabak, B. A. (2013). Putting revenge and forgiveness in an evolutionary context. Behavioral and Brain Sciences, 36, 41–58. doi:10.1017/S0140525X12001513.
McKenna, M. (2012). Directed blame and conversation. In D. J. Coates & N. A. Tognazzini (Eds.), Blame: Its nature and norms (pp. 119–140). New York, NY: Oxford University Press.
MHAT-IV. (2006). Mental Health Advisory Team (MHAT) IV: Operation Iraqi Freedom 05-07 Final report. Washington, DC: Office of the Surgeon, Multinational Force-Iraq; Office of the Surgeon General, United States Army Medical Command.
Mikhail, J. (2007). Universal moral grammar: Theory, evidence and the future. Trends in Cognitive Sciences, 11, 143–152. doi:10.1016/j.tics.2006.12.007.
Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371–378.
Millar, J. (2014). An ethical dilemma: When robot cars must kill, who should pick the victim? Robohub. Robohub.org. Retrieved September 28, 2014, from http://robohub.org/an-ethical-dilemma-when-robot-cars-must-kill-who-should-pick-the-victim/.
Mithen, S. (Ed.). (1998). Creativity in human evolution and prehistory. New York, NY: Taylor & Francis.
Monroe, A. E., Dillon, K. D., & Malle, B. F. (2014). Bringing free will down to earth: People’s psychological concept of free will and its role in moral judgment. Consciousness and Cognition, 27, 100–108. doi:10.1016/j.concog.2014.04.011.
Monroe, A. E., & Malle, B. F. (2010). From uncaused will to conscious choice: The need to study, not speculate about people’s folk concept of free will. Review of Philosophy and Psychology, 1, 211–224.
Monroe, A. E., & Malle, B. F. (2014). Free will without metaphysics. In A. R. Mele (Ed.), Surrounding free will (pp. 25–48). New York, NY: Oxford University Press.
Moor, J. H. (2006). The nature, importance, and difficulty of machine ethics. IEEE Intelligent Systems, 21, 18–21. doi:10.1109/MIS.2006.80.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we know: Verbal reports on mental processes. Psychological Review, 84, 231–259.
Nourbakhsh, I. R. (2013). Robot futures. Cambridge, MA: MIT Press.
Open Roboethics Initiative. (2014a). If death by autonomous car is unavoidable, who should die? Reader poll results.
Open Roboethics Initiative. (2014b). My (autonomous) car, my safety: Results from our reader poll.
Parthemore, J., & Whitby, B. (2013). What makes any agent a moral agent? Reflections on machine consciousness and moral agency. International Journal of Machine Consciousness, 4, 105–129.
Paxton, J. M., Ungar, L., & Greene, J. D. (2012). Reflection and reasoning in moral judgment. Cognitive Science, 36, 163–177. doi:10.1111/j.1551-6709.2011.01210.x.
Petersen, S. (2007). The ethics of robot servitude. Journal of Experimental & Theoretical Artificial Intelligence, 19, 43–54. doi:10.1080/09528130601116139.
Powell, N. L., Derbyshire, S. W. G., & Guttentag, R. E. (2012). Biases in children’s and adults’ moral judgments. Journal of Experimental Child Psychology, 113, 186–193. doi:10.1016/j.jecp.2012.03.006.
Powers, T. M. (2006). Prospects for a Kantian machine. IEEE Intelligent Systems, 21, 46–51. doi:10.1109/MIS.2006.77.
Powers, T. M. (2011). Incremental machine ethics. Robotics & Automation Magazine, IEEE, 18, 51–58. doi:10.1109/MRA.2010.940152.
Pylyshyn, Z. W. (Ed.). (1987). The Robot’s dilemma: The frame problem in artificial intelligence. Norwood, NJ: Ablex.
Ryle, G. (1949). The concept of mind. London: Penguin Press.
Schank, R. C., & Abelson, R. P. (1977). Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Hillsdale, NJ: Erlbaum.
Scheutz, M. (2012). The affect dilemma for artificial agents: Should we develop affective artificial agents? IEEE Transactions on Affective Computing, 3, 424–433.
Scheutz, M., & Crowell, C. R. (2007). The burden of embodied autonomy: Some reflections on the social and ethical implications of autonomous robots. Proceedings of Workshop on Roboethics at ICRA 2007. Rome, Italy.
Scheutz, M., Malle, B. F., & Briggs, G. (2015). Towards morally sensitive action selection for autonomous social robots. The 24th IEEE international symposium on robot and human interactive communication, 2015 RO-MAN. Presented at the 24th IEEE International Symposium on Robot and Human Interactive Communication. (2015). RO-MAN. Japan: Kobe.
Scheutz, M., & Malle, B. F. (2014). “Think and do the right thing”: A plea for morally competent autonomous robots. Presented at the 2014 IEEE Ethics conference, Chicago, IL.
Semin, G. R., & Manstead, A. S. R. (1983). The accountability of conduct: A social psychological analysis. London: Academic Press.
Shaver, K. G. (1985). The attribution of blame: Causality, responsibility, and blameworthiness. New York: Springer.
Sullins, J. P. (2011). Introduction: Open questions in roboethics. Philosophy & Technology, 24, 233. doi:10.1007/s13347-011-0043-6.
Talamadupula, K., Schermerhorn, P., Benton, J., Kambhampati, S., & Scheutz, M. (2011). Planning for agents with changing goals. ICAPS 2011 System Demonstration. Germany: Freiburg.
Tanaka, F., Cicourel, A., & Movellan, J. R. (2007). Socialization between toddlers and robots at an early childhood education center. Proceedings of the National Academy of Sciences, 104, 17954–17958. doi:10.1073/pnas.0707769104.
Tedeschi, J. T., & Reiss, M. (1981). Verbal strategies as impression management. In C. Antaki (Ed.), The psychology of ordinary social behaviour (pp. 271–309). London: Academic Press.
Thiessen, E. D., Kronstein, A. T., & Hufnagle, D. G. (2013). The extraction and integration framework: A two-process account of statistical learning. Psychological Bulletin, 139, 792–814. doi:10.1037/a0030801.
Tomasello, M., & Vaish, A. (2013). Origins of human cooperation and morality. Annual Review of Psychology, 64, 231–255. doi:10.1146/annurev-psych-113011-143812.
Traverso, V. (2009). The dilemmas of third-party complaints in conversation between friends. Journal of Pragmatics, 41, 2385–2399. doi:10.1016/j.pragma.2008.09.047.
Turiel, E. (1983). The development of social knowledge: Morality and convention. Cambridge, England: Cambridge University Press.
Van Berkum, J. J. A., Holleman, B., Nieuwland, M., Otten, M., & Murre, J. (2009). Right or wrong? The brain’s fast response to morally objectionable statements. Psychological Science, 20, 1092–1099. doi:10.1111/j.1467-9280.2009.02411.x.
van Wynsberghe, A. (2013). Designing robots for care: Care centered value-sensitive design. Science and Engineering Ethics, 19, 407–433. doi:10.1007/s11948-011-9343-6.
Veloso, M., Aisen, M., Howard, A., Jenkins, O. C., Mutlu, B., & Scassellati, B. (2012). Human-robot interaction: Japan, South Korea, and China. WTEC Panel Report. Arlington, VA: World Technology Evaluation Center, Inc.
Veruggio, G., Solis, J., & Van der Loos, M. (2011). Roboethics: Ethics applied to robotics. IEEE Robotics Automation Magazine, 18, 21–22. doi:10.1109/MRA.2010.940149.
Voiklis, J., Cusimano, C., & Malle, B. F. (2014). A social-conceptual map of moral criticism. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th annual conference of the cognitive science society (pp. 1700–1705). Austin, TX: Cognitive Science Society.
Walker, M. U. (2006). Moral repair: Reconstructing moral relations after wrongdoing. New York, NY: Cambridge University Press.
Wallach, W. (2010). Robot minds and human ethics: The need for a comprehensive model of moral decision making. Ethics and Information Technology, 12, 243–250. doi:10.1007/s10676-010-9232-8.
Wallach, W., & Allen, C. (2008). Moral machines: Teaching robots right from wrong. New York, NY: Oxford University Press.
Warneken, F., Lohse, K., Melis, A. P., & Tomasello, M. (2011). Young children share the spoils after collaboration. Psychological Science, 22, 267–273. doi:10.1177/0956797610395392.
Weiner, B. (1995). Judgments of responsibility: A foundation for a theory of social conduct. New York, NY: Guilford Press.
Williams, K. D. (2009). Ostracism: A temporal need-threat model. In M. P. Zanna (Ed.), Advances in experimental social psychology (Vol. 41, pp. 275–314). San Diego, CA: Elsevier Academic Press.
Wolpert, D. M., & Flanagan, J. R. (2001). Motor prediction. Current Biology, 11, R729–R732. doi:10.1016/S0960-9822(01)00432-8.
Wright, J. C., & Bartsch, K. (2008). Portraits of early moral sensibility in two children’s everyday conversations. Merrill-Palmer Quarterly, 54, 56–85. doi:10.2307/23096079.
Wyman, E., Rakoczy, H., & Tomasello, M. (2009). Normativity and context in young children’s pretend play. Cognitive Development, 24, 146–155. doi:10.1016/j.cogdev.2009.01.003.
Acknowledgments
This project was partially supported by a grant from the Office of Naval Research (ONR), No. N00014-13-1-0269. The opinions expressed here are my own and do not necessarily reflect the views of ONR. The ideas on moral competence featured in this article have been developed jointly with Matthias Scheutz, Tufts University.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Malle, B.F. Integrating robot ethics and machine morality: the study and design of moral competence in robots. Ethics Inf Technol 18, 243–256 (2016). https://doi.org/10.1007/s10676-015-9367-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10676-015-9367-8