Toward Modeling and Automating Ethical Decision Making: Design, Implementation, Limitations, and Responsibilities

Abstract

One recent priority of the U.S. government is developing autonomous robotic systems. The U.S. Army has funded research to design a metric of evil to support military commanders with ethical decision-making and, in the future, allow robotic military systems to make autonomous ethical judgments. We use this particular project as a case study for efforts that seek to frame morality in quantitative terms. We report preliminary results from this research, describing the assumptions and limitations of a program that assesses the relative evil of two courses of action. We compare this program to other attempts to simulate ethical decision-making, assess possibilities for overcoming the trade-off between input simplification and output reliability, and discuss the responsibilities of users and designers in implementing such programs. We conclude by discussing the implications that this project highlights for the successes and challenges of developing automated mechanisms for ethical decision making.

This is a preview of subscription content, log in to check access.

Fig. 1

References

  1. Anderson SL, Anderson M (2009) How machines can advance ethics. Philosophy Now 72:17–19

    Google Scholar 

  2. Anderson EF, Frith KH, Caspers B (2011) Linking economics and quality: developing an evidence-based nurse staffing tool. Nurs Admin Q 35:53–60

    Article  Google Scholar 

  3. Arkin RC (2007) Governing lethal behavior: embedding ethics in a hybrid deliberative/reactive robot architecture. U.S. Army Research Office Technical Report GIT-GVU-07-11

  4. Baron-Cohen S (2003) The science of evil: on empathy and the origins of cruelty. Basic Books, New York

    Google Scholar 

  5. Beauchamp TL, Childress JF (1979) Principles of biomedical ethics. Oxford University Press, New York

    Google Scholar 

  6. Bell MZ (1985) Why expert systems fail. J Oper Res Soc 36:613–619

    Google Scholar 

  7. Belmont Report (1979) Belmont report—ethical principles and guidelines for the protection of human subjects of research. Available online at http://ohsr.od.nih.gov/guidelines/belmont.html

  8. Center for Modeling, Simulation, and Analysis, and Center for the Management of Science and Technology [CMSA/CMOST] (2010) Developing and calibrating a quantitative metric of evil for use in course of action analysis. Final technical report. AMREDC, SSDD

  9. Dawes RM (1971) A case study of graduate admissions: applications of three principles of human decision making. Am Psychol 26:180–188

    Article  Google Scholar 

  10. Dawes RM (1979) The robust beauty of improper linear models in decision making. Am Psychol 34:571–582

    Article  Google Scholar 

  11. Dawes RM, Corrigan B (1974) Linear models in decision making. Psychol Bull 81:93–106

    Article  Google Scholar 

  12. Dawes RM, Faust D, Meehl PE (1989) Clinical versus actuarial judgment. Science 243:1668–1674

    Article  Google Scholar 

  13. DeMarco JP (2000) Principalism and moral dilemmas: a new principle. J Med Ethics 31:101–105

    Article  Google Scholar 

  14. Dixit AK, Skeath S (2004) Games of strategy, 2nd edn. W.W. Norton & Company, Inc, New York

    Google Scholar 

  15. Fan R (1997) Self-determination vs. family-determination: two incommensurable principles of autonomy: a report from East Asia. Bioethics 11:309–322

    Article  Google Scholar 

  16. Finn P (2011) A future for drones: automated killing. The Washington Post. Retrieved 26 June 2012, from http://www.washingtonpost.com/national/national-security/a-future-for-drones-automated-killing/2011/09/15/gIQAVy9mgK_story.html

  17. Gips J (1995) Towards the ethical robot. In: Ford K, Glymour C, Hayes P (eds) Android epistemology. MIT Press, Cambridge, pp 243–252

    Google Scholar 

  18. Goodwin P, Wright G (2004) Decision analysis for management judgment, 3rd edn. Wiley, West Sussex

    Google Scholar 

  19. Holm S (1995) Not just autonomy—the principles of American biomedical ethics. J Med Ethics 21:332–338

    Article  Google Scholar 

  20. McLaren BM (2003) Extensionally defining principles and cases: an AI model. Artif Intell J 150:145–181

    Article  Google Scholar 

  21. McLaren BM (2005) Lessons in machine ethics from the perspective of two computational models of ethical reasoning. Papers from the AAAI Fall symposium, technical report FS-05-06, pp 70–77

  22. McLaren BM (2006) Computational models of ethical reasoning: challenges, initial steps, and future directions. IEEE Intell Syst 6:2–10

    Google Scholar 

  23. Mental Health Advisory Team (MHAT) IV, Office of the Surgeon General (2006) Final report: operation Iraqi freedom 05-07

  24. Mitchell TO (1969) Observer’s hostility as a factor in judgments of behavior in hostility-provoking situations. Ph.D. Dissertation

  25. Morrow L (2003) Evil: an investigation. Basic Books, New York

    Google Scholar 

  26. Ornstein SM (1987) Computers in battle: a human overview. In: Bellin DB, Chapman G (eds) Computers in battle—will they work?. Harcourt Brace Jovanovich, Inc., Orlando, pp 1–43

    Google Scholar 

  27. Rogerson MD, Gottlieb MC, Handelsman MH, Knapp S, Younggren J (2011) Nonrational processes in ethical decision making. Am Psychol 66:614–623

    Article  Google Scholar 

  28. Rozoff R (2010) Decade of the drone: America’s aerial assassins. Global Res. Retrieved 26 June 2012, from http://globalresearch.ca/index.php?context=va&aid=18027

  29. Sharkey N (2008a) The ethical frontiers of robotics. Science 322:1800–1801

    Article  Google Scholar 

  30. Sharkey N (2008b) Grounds for discrimination: autonomous robot weapons. Rusi Defense Syst 11:86–89

    Google Scholar 

  31. Sullins JP (2010) RoboWarfare: can robots be more ethical than humans on the battlefield? Ethics Inf Technol 12:263–275

    Article  Google Scholar 

  32. Tackett GB (2009) Framework for quantification of evil as a metric for course of action (CoA) analysis. Draft technical report. AMRDEC, RDECOM

  33. U.S. Department of Defense (2007) Unmanned systems roadmap 2007–2032

  34. Welner M (2007) Classifying crimes by severity: from aggravators to depravity. In: Douglass J, Ressler R, Burgess A (eds) A crime classification manual. Jossey-Bass, San Francisco, pp 55–72

    Google Scholar 

  35. Zimbardo PG (2004) A situationist perspective on the psychology of evil: understanding how good people are transformed into perpetrators. In: Miller AG (ed) The social psychology of good and evil. Guilford Press, New York, pp 21–50

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Nicholaos Jones.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Reed, G.S., Jones, N. Toward Modeling and Automating Ethical Decision Making: Design, Implementation, Limitations, and Responsibilities. Topoi 32, 237–250 (2013). https://doi.org/10.1007/s11245-012-9127-x

Download citation

Keywords

  • Decision support
  • Ethical judgment
  • Evil
  • Military
  • Modeling and simulation
  • Robotic systems