Ethical Theory and Moral Practice

, Volume 22, Issue 4, pp 993–1015 | Cite as

What’s Wrong with Designing People to Serve?

  • Bartek ChomanskiEmail author


In this paper I argue, contrary to recent literature, that it is unethical to create artificial agents possessing human-level intelligence that are programmed to be human beings’ obedient servants. In developing the argument, I concede that there are possible scenarios in which building such artificial servants is, on net, beneficial. I also concede that, on some conceptions of autonomy, it is possible to build human-level AI servants that will enjoy full-blown autonomy. Nonetheless, the main thrust of my argument is that, in building such artificial agents, their creators cannot help but evince an objectionable attitude akin to the Aristotelian vice of manipulativeness.


Autonomy Artificial intelligence Manipulativeness Robot ethics 



I have benefitted from discussing the ideas contained in this paper with Arden Ali, John Basl, Kay Mathiesen, Ron Sandler, Ben Yelle, as well I have benefitted from discussing the ideas contained in this paper with Arden Ali, John Basl, Kay Mathiesen, Ron Sandler, Ben Yelle, as well as the students in my Technology and Human Values courses at Northeastern University. I am also grateful to two reviewers for this journal for many insightful comments and suggestions.


  1. Baron M (2003). Manipulativeness. Paper presented at the Proceedings and Addresses of the American Philosophical Association.Google Scholar
  2. Benthall S (2017). Don't fear the reaper: refuting Bostrom's superintelligence argument. arXiv preprint arXiv:1702.08495 Google Scholar
  3. Bloom P, Harris S (2018, It’s Westworld. What’s Wrong With Cruelty to Robots? The New York TimesGoogle Scholar
  4. Borjas G (2009). Immigration. The concise encyclopedia of economics. Retrieved from Accessed 20 Jan 2019.
  5. Boubtane E, Dumont JC, & Rault C (2015). Immigration and economic growth in the OECD countries 1986–2006. Oxford Economic Papers 68(2): 340–360Google Scholar
  6. Brennan J, Jaworski P (2016) Markets without limits: moral virtues and commercial interests. Routledge, New York and LondonGoogle Scholar
  7. Bryson JJ (2010) Robots should be slaves. In: Wilks Y (ed) Close engagements with artificial companions. John Benjamins, Amsterdam, pp 63–74CrossRefGoogle Scholar
  8. Bryson JJ (2018). Patiency is not a virtue: the design of intelligent systems and systems of ethics. Ethics and Information Technology, 20(1), 15-26. Retrieved from CrossRefGoogle Scholar
  9. Bryson JJ, Diamantis ME, & Grant TD (2017). Of, for, and by the people: the legal lacuna of synthetic persons. Artificial Intelligence and Law, 25(3), 273-291. Retrieved from CrossRefGoogle Scholar
  10. Burkeman O (2016). Why you should be nice to your robots. The Guardian. Retrieved from Accessed 20 Jan 2019.
  11. Buss S & Westlund, A. (2018). Personal autonomy. Stanford encyclopedia of philosophy. Retrieved from Accesseed 20 Jan 2019.
  12. Card D (1990) The impact of the Mariel boatlift on the Miami labor market. ILR Rev 43(2):245–257CrossRefGoogle Scholar
  13. Chalmers D (2010) The singularity: a philosophical analysis. J Conscious Stud 17(9–10):7–65Google Scholar
  14. Chomanski B (2018). Massive technological unemployment without Redistribution: A Case for Cautious Optimism. Science and Engineering Ethics. Retrieved from 10.1007/s11948-018-0070-0. Scholar
  15. Danaher J (2019). Welcoming robots into the moral Circle: A Defence of Ethical Behaviourism. Science and Engineering Ethics. Retrieved from 10.1007/s11948-019-00119-x.
  16. Darling K (2014) Extending legal protection to social robots: the effects of anthropomorphism, empathy, and violent behavior towards robotic objects. In: Calo R, Froomkin AM, Kerr I (eds) Robot law. Edward Elgar, Cheltenham, pp 212–232Google Scholar
  17. di Giovanni J, Levchenko, A. A., & Ortega, F. (2015). A global view of cross-border migration. Journal of the European Economic Association, 13(1), 168-202. Retrieved from 10.1111/jeea.12110. CrossRefGoogle Scholar
  18. Floridi L (2013) The ethics of information, (first edition. ed.) edn. Oxford University Press, OxfordGoogle Scholar
  19. Foddy B, Savulescu J (2010) A Liberal account of addiction. Philosophy, Psychiatry, & Psychology 17(1):1–22. Retrieved from CrossRefGoogle Scholar
  20. Foged M, Peri G (2016) Immigrants' effect on native workers: new analysis on longitudinal data. Am Econ J Appl Econ 8(2):1–34CrossRefGoogle Scholar
  21. Frankfurt HG (1971) Freedom of the will and the concept of a person. J Philos 68(1):5–20CrossRefGoogle Scholar
  22. Gunkel DJ (2018) The other question: can and should robots have rights? Ethics Inf Technol 20(2):87–99CrossRefGoogle Scholar
  23. Hakli R, Mäkelä P (2019) Moral responsibility of robots and hybrid agents. Monist 102(2):259–275CrossRefGoogle Scholar
  24. Hanson R (2012) Meet the new conflict, same as the old conflict. J Conscious Stud 19(1–2):119–125Google Scholar
  25. Krugman P, Obstfeld M (2009) International economics: theory and policy. Pearson, LondonGoogle Scholar
  26. LaBossiere M (2017) Testing the moral status of artificial beings; or “I’m going to ask you some questions …”. In: Lin P, Jenkins R, Abney K (eds) Robot ethics 2.0. Oxford University Press, New York, pp 293–306Google Scholar
  27. Levy D (2009) The ethical treatment of artificially conscious robots. Int J Soc Robot 1(3):209–216CrossRefGoogle Scholar
  28. Longhi S, Nijkamp P, Poot J (2005) A meta-analytic assessment of the effect of immigration on wages. J Econ Surv 19(3):451–477CrossRefGoogle Scholar
  29. McDermott D (2012) Response to 'The Singularity' by David Chalmers. J Conscious Stud 19(1–2):167–172Google Scholar
  30. Mele AR (1995) Autonomous agents: from self-control to autonomy. Oxford University Press, New YorkGoogle Scholar
  31. Müller V, Bostrom N (2014) Future progress in artificial intelligence: a survey of expert opinion. In: Müller V (ed) Fundamental issues of artificial intelligence. Springer, BerlinGoogle Scholar
  32. Musiał M (2017) Designing (artificial) people to serve–the other side of the coin. Journal of Experimental & Theoretical Artificial Intelligence 29(5):1087–1097CrossRefGoogle Scholar
  33. Omohundro SM (2008) The basic AI drives. In: Wang P, Goertzel B, Franklin S (eds) Artificial general intelligence, 2008: proceedings of the first AGI conference. IOS Press, Amsterdam, pp 483–492Google Scholar
  34. Ottaviano GI, Peri G (2012) Rethinking the effect of immigration on wages. J Eur Econ Assoc 10(1):152–197CrossRefGoogle Scholar
  35. Peters F, (2019). Cognitive self-management requires the phenomenal registration of intrinsic state properties. Philosophical Studies, 1–23Google Scholar
  36. Petersen S (2011) Designing people to serve. In: Lin P, Bekey G, Abney K (eds) Robot ethics. MIT Press, Cambridge, pp 283–298Google Scholar
  37. Purves D, Jenkins R, Strawser BJ (2015) Autonomous machines, moral judgment, and acting for the right reasons. Ethical Theory Moral Pract 18(4):851–872CrossRefGoogle Scholar
  38. Raz J (1986) The morality of freedom. In: Oxford Oxfordshire; New York: Clarendon press. Press, Oxford UniversityGoogle Scholar
  39. Schneider S (2018) Artificial intelligence, consciousness, and moral status. In: Johnson LSM, Rommelfanger K (eds) Routledge handbook of Neuroethics. Routledge, New YorkGoogle Scholar
  40. Schwitzgebel E, Garza M (2015) A defense of the rights of artificial intelligences. Midwest Studies in Philosophy 39(1):98–119CrossRefGoogle Scholar
  41. Sparrow R (1999) The ethics of terraforming. Environ Ethics 21(3):227–245CrossRefGoogle Scholar
  42. Thaler RH, Sunstein CR (2008) Nudge : improving decisions about health, wealth, and happiness. Yale University Press, New HavenGoogle Scholar
  43. Turner J (2019) Robot rules. Pagrave Macmillan, ChaimCrossRefGoogle Scholar
  44. Walker M (2006) A moral paradox in the creation of artificial intelligence: Mary Poppins 3000s of the world unite! In: Metzler T (ed) Human implications of human-robot interaction: papers from the AAAI workshop. AAAI Press, Menlo Park, pp 23–28Google Scholar
  45. Walker M (2016) Free money for all : a basic income guarantee solution for the twenty-first century. Palgrave Macmillan, New YorkCrossRefGoogle Scholar
  46. Wellman CH (2015). Immigration. The Stanford Encyclopedia of Philosophy Summer 2015. Retrieved from Accessed 1 May 2019.
  47. Ziesche S, Yampolskiy R (2019) Towards AI welfare science and policies. Big Data and Cognitive Computing 3(2)CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of Philosophy and ReligionNortheastern UniversityBostonUSA

Personalised recommendations