, Volume 34, Issue 4, pp 907–920 | Cite as

The problem of superintelligence: political, not technological

  • Wolfhart TotschnigEmail author
Open Forum


The thinkers who have reflected on the problem of a coming superintelligence have generally seen the issue as a technological problem, a problem of how to control what the superintelligence will do. I argue that this approach is probably mistaken because it is based on questionable assumptions about the behavior of intelligent agents and, moreover, potentially counterproductive because it might, in the end, bring about the existential catastrophe that it is meant to prevent. I contend that the problem posed by a future superintelligence will likely be a political problem, that is, one of establishing a peaceful form of coexistence with other intelligent agents in a situation of mutual vulnerability, and not a technological problem of control.


Superintelligence Existential risk Source of normativity 



I would like to thank Aïcha Liviana Messina for encouraging me to present a first draft of this paper at a conference on cosmopolitanism at the Universidad Diego Portales in December 2015, and Paula Boddington, Peter Millican, and Michael Wooldridge for inviting me to deliver a more developed draft at a workshop on the ethics of artificial intelligence at the 25th International Joint Conference on Artificial Intelligence. I would also like to thank Karoline Feyertag, Juan Manuel Garrido, Yehida Mendoza, Simon Mussell, Juan Ormeño Karzulovic, Michał Prządka, Nicolò Sibilla, Jaan Tallinn, Szymon Toruńczyk, Johanna Totschnig, Michael Totschnig, Christoph Weiss, and the reviewers for AI & Society for their comments on subsequent versions of this paper. Lastly, I am grateful to Stefan Sorgner for giving me the opportunity to present the paper at the 9th Beyond Humanism Conference. Without their encouragement and critique, this project would not have come to fruition.


  1. Bostrom N (1998) How long before superintelligence? Accessed 22 Aug 2016.
  2. Bostrom N (2002) Existential risks: analyzing human extinction scenarios and related hazards. J Evol Technol 9(1). Accessed 24 April 2016.
  3. Bostrom N (2003) Ethical issues in advanced artificial intelligence. Accessed 9 March 2016.
  4. Bostrom N (2014) Superintelligence: paths, dangers, strategies. Oxford University Press, OxfordGoogle Scholar
  5. Butler S (1872) 2002. Erewhon. Dover Publications, New YorkGoogle Scholar
  6. Chalmers DJ (2010) The singularity: a philosophical analysis. J Conscious Stud 17(9–10):7–65Google Scholar
  7. Chalmers DJ (2012) The singularity: a reply to commentators. J Conscious Stud 19(7–8):141–167Google Scholar
  8. de Garis H (1990) The 21st century artilect: moral dilemmas concerning the ultra intelligent machine. Rev Int Philos 44(172):131–138Google Scholar
  9. Dreyfus HL (1972) What computers can’t do: a critique of artificial reason. Harper & Row, New YorkGoogle Scholar
  10. Dreyfus HL (1992) What computers still can’t do: a critique of artificial reason. The MIT Press, CambridgeGoogle Scholar
  11. Egan G (1997) Diaspora. Millennium, LondonGoogle Scholar
  12. Future of Life Institute (2015) Autonomous weapons: an open letter from AI & robotics researchers, July 28. Accessed 7 April 2016.
  13. Geraci RM (2008) Apocalyptic AI: religion and the promise of artificial intelligence. J Am Acad Relig 76(1):138–166MathSciNetCrossRefGoogle Scholar
  14. Good IJ (1965) Speculations concerning the first ultraintelligent machine. In: Alt FL, Rubinoff M (eds) Advances in computers, vol 6. Academic, New York, pp 31–88Google Scholar
  15. Hobbes T (1651) 1994. Leviathan. Curley E (ed). Hackett Publishing, IndianapolisGoogle Scholar
  16. Kant I (1785) 1998. Groundwork of the metaphysics of morals. Gregor M (ed). Cambridge University Press, CambridgeGoogle Scholar
  17. Kant I (1788) 2015. Critique of practical reason. Gregor M (ed). Cambridge University Press, CambridgeGoogle Scholar
  18. Kornai A (2016) Bounding the impact of artificial general intelligence. In: Müller VC (ed) Risks of artificial intelligence. CRC Press, Boca Raton, pp 179–211Google Scholar
  19. Kurzweil R (1999) The age of spiritual machines: when computers exceed human intelligence. Viking, New YorkGoogle Scholar
  20. Kurzweil R (2005) The singularity is near: when humans transcend biology. Penguin, New YorkGoogle Scholar
  21. Lucas JR (1961) Minds, machines and Gödel. Philosophy 36(137):112–127CrossRefGoogle Scholar
  22. Moravec H (1999) Robot: Mere machine to transcendent mind. Oxford University Press, OxfordGoogle Scholar
  23. Muehlhauser L, Helm L (2012) The singularity and machine ethics. In: Eden AH, Moor JH, Søraker JH, Steinhart E (eds) Singularity hypotheses: a scientific and philosophical assessment. Springer, Berlin, pp 101–125CrossRefGoogle Scholar
  24. Omohundro SM (2008) The nature of self-improving artificial intelligence. Accessed 18 Nov 2016.
  25. Omohundro SM (2012) Rational artificial intelligence for the greater good. In: Eden AH, Moor JH, Søraker JH, Steinhart E (eds) Singularity hypotheses: a scientific and philosophical assessment. Springer, Berlin, pp 161–176CrossRefGoogle Scholar
  26. Omohundro SM (2016) Autonomous technology and the greater human good. In: Müller VC (ed) Risks of artificial intelligence. CRC Press, Boca Raton, pp 9–27Google Scholar
  27. Penrose R (1991) The Emperor’s new mind: concerning computers, minds, and the laws of physics. Penguin, New YorkzbMATHGoogle Scholar
  28. Searle JR (1980) Minds, brains, and programs. Behav Brain Sci 3(3):417–424CrossRefGoogle Scholar
  29. Simmons D (1989) Hyperion. Doubleday, New YorkGoogle Scholar
  30. Turing AM (1950) Computing machinery and intelligence. Mind 59(236):433–460MathSciNetCrossRefGoogle Scholar
  31. Turing AM (1996) Intelligent machinery, a heretical theory. Philos Math 4(3):256–260MathSciNetCrossRefGoogle Scholar
  32. Vinge V (1992) A fire upon the deep. Tor Books, New YorkGoogle Scholar
  33. Vinge V (1993) The coming technological singularity: how to survive in the post-human era. Accessed March 9, 2016.
  34. Walker MA (2002) Prolegomena to any future philosophy. J Evol Technol 10(1). Accessed 1 Nov 2016.
  35. Wallach W, Allen C (2009) Moral machines: teaching robots right from wrong. Oxford University Press, OxfordCrossRefGoogle Scholar
  36. Wiener N (1960) Some moral and technical consequences of automation. Science 131(3410):1355–1358CrossRefGoogle Scholar
  37. Yampolskiy RV (2013) What to do with the singularity paradox? In: Müller VC (ed) Philosophy and theory of artificial intelligence. Springer, Berlin, pp 397–413CrossRefGoogle Scholar
  38. Yampolskiy RV (2016) Utility function security in artificially intelligent agents. In: Müller VC (ed) Risks of artificial intelligence. CRC Press, Boca Raton, pp 115–140Google Scholar
  39. Yampolskiy RV, Fox J (2012) Artificial general intelligence and the human mental model. In: Eden AH, Moor JH, Søraker JH, Steinhart E (eds) Singularity hypotheses: A scientific and philosophical assessment. Springer, Berlin, pp 129–145CrossRefGoogle Scholar
  40. Yampolskiy RV, Fox J (2013) Safety engineering for artificial general intelligence. Topoi 32(2):217–226Google Scholar
  41. Yudkowsky E (2001) Creating friendly AI 1.0: the analysis and design of benevolent goal architectures. The Singularity Institute, San FranciscoGoogle Scholar
  42. Yudkowsky E (2002) The AI-box experiment. Accessed 9 March 2016.
  43. Yudkowsky E (2008) Artificial intelligence as a positive and negative factor in global risk. In: Bostrom N, Ćirković MM (eds) Global catastrophic risks. Oxford University Press, Oxford, pp 308–345Google Scholar
  44. Yudkowsky E (2011) Complex value systems in friendly AI. In: Schmidhuber J, Thórisson KR, Looks M (eds) Artificial general intelligence. Springer, Berlin, pp 388–393CrossRefGoogle Scholar
  45. Yudkowsky E (2012) Friendly artificial intelligence. In: Eden AH, Moor JH, Søraker JH, Steinhart E (eds) Singularity hypotheses: a scientific and philosophical assessment. Springer, Berlin, pp 181–193CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd. 2017

Authors and Affiliations

  1. 1.Universidad Diego PortalesSantiagoChile

Personalised recommendations