Ethical Operating Systems

  • Naveen Sundar Govindarajulu
  • Selmer Bringsjord
  • Atriya Sen
  • Jean-Claude Paquin
  • Kevin O’Neill
Part of the Philosophical Studies Series book series (PSSP, volume 133)


A well-ingrained and recommended engineering practice in safety-critical software systems is to separate safety concerns from other aspects of the system. Along these lines, there have been calls for operating systems (or computing substrates, termed ethical operating systems) that implement ethical controls in an ethical layer separate from, and not amenable to tampering by, developers and modules in higher-level intelligence or cognition layers. There have been no implementations that demonstrate such a marshalling of ethical principles into an ethical layer. To address this, we present three different tracks for implementing such systems, and offer a prototype implementation of the third track. We end by addressing objections to our approach.


Ethical machines Ethical operating system Deontic cognitive event calculus Ethical layer Doctrine of double effect 



We are indebted to seven anonymous reviewers (of the core of the present version, as well as its predecessor) for insightful comments, suggestions, and objections. In addition, we are grateful to ONR for its support of making morally competent machines, and to AFOSR for its support of our pursuit of computational intelligence in machines, on the strength of novel modes of machine reasoning. Finally, without the energy, passion, intelligence, and wisdom of both Giuseppe Primiero and Liesbeth De Mol, any progress we have made in the direction of ethical OSs would be non-existent.


  1. Annas, J. 2011. Intelligent virtue. Oxford: Oxford University Press.CrossRefGoogle Scholar
  2. Anscombe, G. 1958. Modern moral philosophy. Philosophy 33(124): 1–19.CrossRefGoogle Scholar
  3. Arkin, R. 2009. Governing lethal behavior in autonomous robots. New York: Chapman and Hall/CRC.CrossRefGoogle Scholar
  4. Arkoudas, K., K. Zee, V. Kuncak, and M. Rinard. 2004. Verifying a file system implementation. In Sixth International Conference on Formal Engineering Methods (ICFEM’04), Lecture notes in computer science (LNCS), vol. 3308, 373–390. Seattle: Springer.Google Scholar
  5. Arkoudas, K., S. Bringsjord, and P. Bello. 2005. Toward ethical robots via mechanized deontic logic. In Machine Ethics: Papers from the AAAI Fall Symposium; FS–05–06, 17–23. Menlo Park: American Association for Artificial Intelligence. Google Scholar
  6. Banker, S. 2016. Using big data and predictive analytics to predict which truck drivers will have an accident. Available at:
  7. Bentzen, M.M. 2016. The principle of double effect applied to ethical dilemmas of social robots. In Frontiers in Artificial Intelligence and Applications, Proceedings of Robophilosophy 2016/TRANSOR 2016, 268–279. Amsterdam: IOS Press.Google Scholar
  8. Berreby, F., G. Bourgne, and J.-G. Ganascia. 2015. Modelling moral reasoning and ethical responsibility with logic programming. In Logic for programming, artificial intelligence, and reasoning, 532–548. Berlin/Heidelberg: Springer.CrossRefGoogle Scholar
  9. Bojarski, M., D.D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L.D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. 2016. End to end learning for self-driving cars. CoRR abs/1604.07316.
  10. Bon´er, J. 2010. Introducing Akka—simpler scalability, fault-tolerance, concurrency & remoting through actors.
  11. Boolos, G.S., J.P. Burgess, and R.C. Jeffrey. 2003. Computability and logic, 4th edn. Cambridge: Cambridge University Press.Google Scholar
  12. Bringsjord, S. 2015a. A 21st-century ethical hierarchy for humans and robots: \(\mathcal {EH}\). In A World With Robots: International Conference on Robot Ethics (ICRE 2015), ed. I. Ferreira, J. Sequeira, M. Tokhi, E. Kadar, and G. Virk, 47–61. Berlin: Springer.Google Scholar
  13. Bringsjord, S. 2015b. A vindication of program verification. History and philosophy of logic 36(3): 262–277.CrossRefGoogle Scholar
  14. Bringsjord, S. 2016. Can phronetic robots be engineered by computational logicians? In Proceedings of Robophilosophy/TRANSOR 2016, ed. J. Seibt, M. Nørskov, and S. Andersen, 3–6. Amsterdam: IOS Press.Google Scholar
  15. Bringsjord, S., and N.S. Govindarajulu. 2012. Given the Web, what is intelligence, really? Metaphilosophy 43(4): 361–532.CrossRefGoogle Scholar
  16. Bringsjord, S., and J. Taylor. 2012. The divine-command approach to robot ethics. In Robot ethics: The ethical and social implications of robotics, ed. P. Lin, G. Bekey, and K. Abney, 85–108. Cambridge: MIT Press.Google Scholar
  17. Bringsjord, S., and A. Sen. 2016. On creative self-driving cars: Hire the computational logicians, fast. Applied Artificial Intelligence 30: 758–786.CrossRefGoogle Scholar
  18. Bringsjord, S., K. Arkoudas, and P. Bello. 2006. Toward a general logicist methodology for engineering ethically correct robots. IEEE Intelligent Systems 21(4): 38–44.CrossRefGoogle Scholar
  19. Bringsjord, S., J. Taylor, A. Shilliday, M. Clark, and K. Arkoudas. 2008. Slate: An argument-centered intelligent assistant to human reasoners. In Proceedings of the 8th International Workshop on Computational Models of Natural Argument (CMNA 8)’, ed. F. Grasso, N. Green, R. Kibble, and C. Reed, 1–10. Patras: University of Patras.Google Scholar
  20. Bringsjord, S., N. Govindarajulu, D. Thero, and M. Si. 2014. Akratic robots and the computational logic thereof. In Proceedings of ETHICS 2014, (2014 IEEE Symposium on Ethics in Engineering, Science, and Technology), 22–29, Chicago.
  21. Chisholm, R. 1982. Supererogation and offence: A conceptual scheme for ethics. In Brentano and Meinong studies, ed. R. Chisholm, 98–113. Atlantic Highlands: Humanities Press.Google Scholar
  22. Dijkstra, E.W. 1982. On the role of scientific thought. In Selected writings on computing: A personal perspective, 60–66. New York: Springer.CrossRefGoogle Scholar
  23. Feldman, F. 1978. Introductory ethics. Englewood Cliffs: Prentice-Hall.Google Scholar
  24. Flatt, M., R. Findler, S. Krishnamurthi, and M. Felleisen. 1999. Programming languages as operating systems (or revenge of the son of the Lisp machine). In Proceedings of the International Conference on Functional Programming (ICFP 1999).
  25. Ganascia, J.-G. 2007. Modeling ethical rules of lying with answer set programming. Ethics and Information Technology 9: 39–47.CrossRefGoogle Scholar
  26. Ganascia, J.-G. 2015. Non-monotonic resolution of conflicts for ethical reasoning. In A construction manual for robots’ ethical systems: Requirements, methods, implementations, ed. R. Trappl, 101–118. Basel: Springer.CrossRefGoogle Scholar
  27. Govindarajulu, N.S. 2010. Common Lisp actor system. See also:
  28. Govindarajulu, N.S., and S. Bringsjord. 2015. Ethical regulation of robots must be embedded in their operating systems. In A construction manual for robots’ ethical systems: Requirements, methods, implementations, ed. R. Trappl, 85–100. Basel: Springer.CrossRefGoogle Scholar
  29. Govindarajulu, N.S., and S. Bringsjord. 2017. On automating the doctrine of double effect. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17’, ed. C. Sierra, 4722–4730, Melbourne.Google Scholar
  30. Hursthouse, R., and G. Pettigrove. 2003/2016. Virtue ethics. In The stanford encyclopedia of philosophy, Metaphysics research lab, ed. E. Zalta. Stanford University.
  31. Hutter, M. 2005. Universal artificial intelligence: Sequential decisions based on algorithmic probability. New York: Springer.CrossRefGoogle Scholar
  32. Johnson, G. 2016. Argument & inference: An introduction to inductive logic. Cambridge: MIT Press.Google Scholar
  33. Kwiatkowska, M., G. Norman, and D. Parker. 2011. PRISM 4.0: Verification of probabilistic real-time systems. In International Conference on Computer Aided Verification, 585–591. Berlin: Springer.CrossRefGoogle Scholar
  34. McIntyre, A. 2014. Doctrine of double effect. In The stanford encyclopedia of philosophy, winter 2014 edn, Metaphysics Research Lab, ed. E.N. Zalta. Stanford University.Google Scholar
  35. McKinsey, J., A. Sugar, and P. Suppes. 1953. Axiomatic foundations of classical particle mechanics. Journal of Rational Mechanics and Analysis 2: 253–272.Google Scholar
  36. Naumowicz, A., and A. Kornilowicz. 2009. A brief overview of Mizar. In Theorem proving in higher order logics, Lecture notes in computer science (LNCS), vol. 5674, ed. S. Berghofer, T. Nipkow, C. Urban, and M. Wenzel, 67–72. Berlin: Springer.CrossRefGoogle Scholar
  37. Pereira, L. M., and A. Saptawijaya. 2016a. Counterfactuals, logic programming and agent morality. In Logic, argumentation and reasoning, ed. S. Rahman and J. Redmond, 85–99. Berlin: Springer.Google Scholar
  38. Pereira, L., and A. Saptawijaya. 2016b. Programming machine ethics. Berlin: Springer.CrossRefGoogle Scholar
  39. Ramos, S., S.K. Gehrig, P. Pinggera, U. Franke, and C. Rother. 2016. Detecting unexpected obstacles for self-driving cars: Fusing deep learning and geometric modeling. CoRR, abs/1612.06573.
  40. Russell, S., and P. Norvig. 2009. Artificial intelligence: A modern approach, 3rd edn. Upper Saddle River: Prentice Hall.Google Scholar
  41. Varela, C.A. 2013. Programming distributed computing systems: A foundational approach. MIT Press.
  42. Varela, C., and G. Agha. 2001. Programming dynamically reconfigurable open systems with salsa. ACM SIGPLAN Notices, 36(12): 20–34.CrossRefGoogle Scholar
  43. Vaughan, R.T., B.P. Gerkey, and A. Howard. 2003. On device abstractions for portable, reusable robot code. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453), Las Vegas, vol. 3, 2421–2427.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Naveen Sundar Govindarajulu
    • 1
  • Selmer Bringsjord
    • 2
  • Atriya Sen
    • 3
  • Jean-Claude Paquin
    • 4
  • Kevin O’Neill
    • 4
  1. 1.Department of Cognitive Science, Department of Computer Science, Rensselaer AI & Reasoning (RAIR) LabRPITroyUSA
  2. 2.Department of Computer Science, Department of Cognitive Science, RAIR Lab, Lally School of ManagementRPITroyUSA
  3. 3.RAIR Lab, Department of Computer ScienceRPITroyUSA
  4. 4.RAIR LabRPITroyUSA

Personalised recommendations