pp 1–17 | Cite as

Classification of global catastrophic risks connected with artificial intelligence

  • Alexey TurchinEmail author
  • David Denkenberger
Original Article


A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various levels of AI development, namely, (1) before it starts self-improvement, (2) during its takeoff, when it uses various instruments to escape its initial confinement, or (3) after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level.


Artificial intelligence Global risks Military drones Superintelligence Existential risk 



We would like to thank Roman Yampolskiy and Seth Baum for their interesting ideas in this article. This article represents views of the authors and does not necessarily represent the views of the Global Catastrophic Risk Institute or the Alliance to Feed the Earth in Disasters. No external sources of funding were used for this work.


  1. Alexander S (2016) Ascended economy? Star Slate Codex. Accessed 27 Apr 2018
  2. Anderson M (2017) RethinkX: self-driving electric cars will dominate roads by 2030. In: IEEE Spectrum: technology, engineering and science news. Accessed 17 Jul 2017
  3. (2017) Artificial intelligence startups. Accessed 27 Apr 2018
  4. Armstrong S (2017) Good and safe uses of AI Oracles. ArXiv171105541 CsGoogle Scholar
  5. Auerbach D (2014) The Most Terrifying Thought Experiment of All Time. In: Slate. Accessed 27 Apr 2018
  6. Baker BH (2000) The gray matter: the forgotten story of the telephone. Telepress, Kent, WAGoogle Scholar
  7. Bardi U (2008) The Universal Mining Machine. Accessed 27 Apr 2018
  8. Barrett AM, Baum SD (2017) A model of pathways to artificial superintelligence catastrophe for risk and decision analysis. J Exp Theor Artif Intell 29:397–414CrossRefGoogle Scholar
  9. BBC (2017) Cyber-attack: europol says it was unprecedented in scale—BBC News. Accessed 17 Jul 2017
  10. Bender J (2014) Russia may still have an automated nuclear launch system aimed across the northern hemisphere. In: Bus. Insid. Accessed 17 Jul 2017
  11. Blair BG (2011) The logic of accidental nuclear war. Brookings Institution Press, Washington, DCGoogle Scholar
  12. Boles KS, Kannan K, Gill J et al (2017) Digital-to-biological converter for on-demand production of biologics. Nat Biotechnol 35:672–675 2017CrossRefGoogle Scholar
  13. Bostrom N (2002) Existential risks: analyzing human extinction scenarios and related hazards. J Evol Technol 9(1):1–30Google Scholar
  14. Bostrom N (2003a) Astronomical waste: The opportunity cost of delayed technological development. Utilitas 15:308–314CrossRefGoogle Scholar
  15. Bostrom N (2003b) Are you living in a computer simulation? Publ Philos Q 53(211):243–255CrossRefGoogle Scholar
  16. Bostrom N (2006) What is a singleton. Linguist Philos Investig 5:48–54Google Scholar
  17. Bostrom N (2009) Pascal’s mugging. Analysis 69(3):443–445MathSciNetCrossRefGoogle Scholar
  18. Bostrom N (2011) Infinite ethics. Anal Metaphys 9–59Google Scholar
  19. Bostrom N (2014) Superintelligence. Oxford University Press, OxfordGoogle Scholar
  20. Bradbury RJ (2001) Matrioshka brains. preprint.
  21. Carrigan RA Jr (2006) Do potential SETI signals need to be decontaminated? Acta Astronaut 58:112–117CrossRefGoogle Scholar
  22. Chalmers DJ (2002) Does conceivability entail possibility? In: Gendler T, Hawthorne J (eds) Conceivability possibility. Oxford University Press, New York pp 145–200Google Scholar
  23. Chiew KL, Yong KSC, Tan CL (2018) A survey of phishing attacks: their types, vectors and technical approaches. Expert Syst Appl 106:1–20CrossRefGoogle Scholar
  24. Christiano P (2016) Prosaic AI alignment. Accessed 27 Apr 2018
  25. Clavero M, García-Berthou E (2005) Invasive species are a leading cause of animal extinctions. Trends Ecol Evol 20:110CrossRefGoogle Scholar
  26. Cole DD, Denkenberger D, Griswold M et al (2016) Feeding everyone if industry is disabled. In: Proceedings of the 6th international disaster and risk conference. Davos, SwitzerlandGoogle Scholar
  27. Critch A (2017) Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making (arXiv:1701.01302)Google Scholar
  28. Daniel M (2017) S-risks: why they are the worst existential risks, and how to prevent them (EAG Boston 2017). Accessed 27 Apr 2018
  29. Dennett DC (1978) Why you can’t make a computer that feels pain. Synthese 38:415–456CrossRefGoogle Scholar
  30. Ellison H (1967) I have no mouth, and i must scream. Galaxy Publishing Corp, New YorkGoogle Scholar
  31. Enserink M (2011) Scientists brace for media storm around controversial flu studies. In: Sciencemag. Accessed 27 Apr 2018
  32. Freitas R (2000) Some limits to global ecophagy by biovorous nanoreplicators, with public policy recommendations. Foresight Institute Technical ReportGoogle Scholar
  33. Future of Life Institute (2016) Accidental nuclear war: a timeline of close calls. Accessed 4 Nov 2017
  34. Futureworld (2013) Airplane “crashes” as hacker gets control. In: Futureworld. Accessed 27 Apr 2018
  35. Gildert S (2011) Why “computronium” is really “unobtanium” IO9. Accessed 27 Apr 2018
  36. Goertzel B (2012) Should humanity build a global ai nanny to delay the singularity until it’s better understood? J Conscious Stud 19(1–2):96–111Google Scholar
  37. Grace K, Salvatier J, Dafoe A et al (2017) When will AI exceed human performance? evidence from AI experts. (arXiv:1705.08807 [cs.AI])Google Scholar
  38. Granoff J (2016) Donald trump is an existential threat to America and the world. TimeGoogle Scholar
  39. Gwern (2016) Why tool AIs want to be agent AIs.
  40. Hanson R (2008) Catastrophe, social collapse, and human extinction. In: Bostrom N, Cirkovic MM (eds) Global catastrophic risks. Oxford University Press, Oxford, p 554Google Scholar
  41. Hanson R (2016) The age of Em: work, love, and life when robots rule the earth. Oxford University Press, OxfordGoogle Scholar
  42. Hines N (2016) Neural implants could let hackers hijack your brain. In: Inverse. Accessed 17 Jul 2017
  43. Hume D (1739) A treatise of human nature. Oxford: Clarendon Press, London, UKGoogle Scholar
  44. Hutter M (2000) A theory of universal artificial intelligence based on algorithmic complexity. ArXiv Prepr Cs0004001Google Scholar
  45. Jenkins A (2018) Uber may not be to blame for self-driving car death in Arizona. Fortune, New YorkGoogle Scholar
  46. Joy B (2000) Why the future doesn’t need us. Wired, San Francisco, CAGoogle Scholar
  47. Kahn H (1959) On thermonuclear war. Princeton University Press, PrincetonGoogle Scholar
  48. Kardashev NS (1985) On the inevitability and the possible structures of supercivilizations. Reidel Publishing Co., Dordrecht, pp 497–504Google Scholar
  49. Karpathy A (2015) The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy Blog.
  50. Kushner D (2013) The real story of stuxnet. IEEE Spectr 50:48–53CrossRefGoogle Scholar
  51. LaVictoire P, Fallenstein B, Yudkowsky E et al (2014) Program equilibrium in the prisoner’s dilemma via Löb’s theorem. MIRIGoogle Scholar
  52. LaVictorie P (2015) An Introduction to Löb’s Theorem in MIRI Research. MIRI, San Francisco CA. Google Scholar
  53. Lem S (1961) Return from the stars. Houghton Mifflin Harcourt, Boston, USGoogle Scholar
  54. Lem S (1963) Summa technologiae. Suhrkamp, Berlin, GermanyGoogle Scholar
  55. Lem S (1973) The Invincible: science fiction. Sidgwick & Jackson, London, UKGoogle Scholar
  56. Lenat DB, Brown JS (1984) Why AM and EURISKO appear to work. Artif Intell 23:269–294CrossRefGoogle Scholar
  57. LoPucki LM (2017) Algorithmic ENTITIES. Social Science Research Network, RochesterGoogle Scholar
  58. Lubin G (2016) Data reveals the 20 most popular TV shows of 2016. Business InsiderGoogle Scholar
  59. Mennen A (2017) Existential risk from AI without an intelligence explosion.
  60. Menzel C (2017) Actualism. In: Zalta EN (ed) The stanford encyclopedia of philosophy, 2014th edn. Metaphysics Research Lab, Stanford University, StanfordGoogle Scholar
  61. Meuhlhauser L (2014) How big is the field of artificial intelligence? (initial findings). Accessed 27 Apr 2018
  62. Muehlhauser L (2011) Intelligence explosion FAQ. Accessed 27 Apr 2018
  63. Mullin G (2017) What is the Blue Whale suicide challenge, how many deaths has the game been linked to so far and is it in the UK? TheSunGoogle Scholar
  64. Oberhaus D (2017) Watch ‘Slaughterbots’, a warning about the future of killer bots. In: Motherboard. Accessed 17 Dec 2017
  65. Omohundro S (2008) The basic AI drives. In: Wang P, Goertzel B, Franklin S (eds) Proceedings of the 2008 conference on Artificial General Intelligence 2008: proceedings of the First AGI Conference. IOS Press Amsterdam, The NetherlandsGoogle Scholar
  66. Orwell G (1948) 1984. Houghton Mifflin Harcourt, Boston, USGoogle Scholar
  67. Pinker S (2011) The better angels of our nature: The decline of violence in history and its causes. Penguin, LondonGoogle Scholar
  68. Reason J (2000) Human error: models and management. BMJ 320:768–770CrossRefGoogle Scholar
  69. Russell S (2017) 3 principles for creating safer AI. Accessed 27 Apr 2018
  70. Saito T, Angles J (2013) Hikikomori: adolescence without end. Univesity Of Minnesota Press, MinnesotaGoogle Scholar
  71. Sarma GP, Hay NJ (2016) Mammalian value systems. (arXiv:1607.08289 [cs.AI]) Google Scholar
  72. Schneier B (2017) Perspective | The next ransomware attack will be worse than WannaCry. Wash, PostGoogle Scholar
  73. Shakirov V (2016) Review of state-of-the-arts in artificial intelligence with application to AI safety problem. (ArXiv Prepr ArXiv160504232) Google Scholar
  74. Shulman C (2010) Omohundro’s “basic AI drives” and catastrophic risks. Accessed 27 Apr 2018
  75. Shulman C (2011) Arms races and intelligence explosions. Singularity Hypotheses. Springer, New YorkGoogle Scholar
  76. Sotala K (2016) Decisive strategic advantage without a hard takeoff. Accessed 27 Apr 2018
  77. Sotala K (2017) Disjunctive AI scenarios: Individual or collective takeoff? Accessed 27 Apr 2018
  78. Sotala K, Yampolskiy R (2014) Responses to catastrophic AGI risk: a survey. Phys Scr 90:018001CrossRefGoogle Scholar
  79. Srugatsky N, Strugatsky B (1985) The time wanderers. Richardson & Steirman, New York, USGoogle Scholar
  80. Strugatsky A, Strugatsky B (1976) The final circle of paradise, Translated by Leonid Renen. DAW, New YorkGoogle Scholar
  81. Taylor A (2017) Flying around the world in a solar powered plane—the AtlanticGoogle Scholar
  82. The Telegraph (2009) Russian spacecraft landed on moon hours before Americans. The telegraph. Accessed 27 Apr 2018
  83. Torres P (2014) Why running simulations may mean the end is near. Accessed 27 Apr 2018
  84. Torres P (2016) Problems with defining an existential risk. IEET. Accessed 27 Apr 2018
  85. Turchin A (2018) The risks connected with possibility of finding alien AI code during SETI. Rev J Br Interplanet Soc. Manuscript,
  86. Turchin A, Denkenberger D (2017) Levels of self-improvement. Manuscript, University of Louisville, TNGoogle Scholar
  87. Turchin A, Denkenberger D (2018a) Military AI as convergent goal of the self-improving AI. In: Yampolskiy R (ed) Artificial intelligence safety and security. CRC Press, Baca RatonGoogle Scholar
  88. Turchin A, Denkenberger D (2018b) Could slaughterbots wipe out humanity? Assessment of the global catastrophic risk posed by autonomous weapons. ManuscriptGoogle Scholar
  89. Turchin A, Green B, Denkenberger D (2017) multiple simultaneous pandemics as most dangerous global catastrophic risk connected with bioweapons and synthetic biology. Rev Health SecurGoogle Scholar
  90. Turing AM (1937) On computable numbers, with an application to the Entscheidungsproblem. Proc Lond Math Soc 2:230–265MathSciNetCrossRefzbMATHGoogle Scholar
  91. Velicovich B (2017) I could kill you with a consumer drone. Defense one, Washington, DCGoogle Scholar
  92. Watkins J (2016) “Shut up and dance”—“Black mirror” seriesGoogle Scholar
  93. Wei D (2013) Outside view(s) and MIRI’s FAI endgame. Accessed 27 Apr 2018
  94. Wootson J (2017) Elon Musk doesn’t think we’re prepared to face humanity’s biggest threat: artificial intelligence. Wash, PostGoogle Scholar
  95. Yampolskiy R (2014) Utility function security in artificially intelligent agents. J Exp Theor Artif Intell JETAI 373–389.
  96. Yampolskiy R (2015a) Artificial superintelligence: a futuristic approach. CRC Press, Boca RatonGoogle Scholar
  97. Yampolskiy R (2015b) Taxonomy of pathways to dangerous AI. (ArXiv Prepr ArXiv151103246) Google Scholar
  98. Yampolskiy R, Spellchecker M (2016) artificial intelligence safety and cybersecurity: a timeline of AI failures. (ArXiv Prepr ArXiv161007997) Google Scholar
  99. Yudkowsky E (2001) Creating friendly AI 1.0: the analysis and design of benevolent goal architectures. MIRI, San Francisco, CA, pp 1–282Google Scholar
  100. Yudkowsky E (2002) The AI-Box Experiment. Accessed 27 Apr 2018
  101. Yudkowsky E (2003) HUMOR: friendly AI critical failure table. Accessed 27 Apr 2018
  102. Yudkowsky E (2004) Coherent extrapolated volition. Accessed 27 Apr 2018
  103. Yudkowsky E (2008) Artificial intelligence as a positive and negative factor in global risk, in global catastrophic risks. Oxford University Press, OxfordGoogle Scholar
  104. Yudkowsky E (2015) From AI to zombies. MIRI, San Francisco, CAGoogle Scholar
  105. Yudkowsky E (2017) Comment on paper clip maximiser scenario. Accessed 27 Apr 2018
  106. Yudkowsky E, Hanson R (2008) The Hanson-Yudkowsky AI-foom debate. In: MIRI Technical reportGoogle Scholar
  107. Yudkowsky E, Herreshoff M (2013) Tiling agents for self-modifying AI, and the Löbian obstacle. Early Draft MIRIGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Science for Life Extension FoundationMoscowRussia
  2. 2.Global Catastrophic Risk Institute (GCRI)Tennessee State University, Alliance to Feed the Earth in Disasters (ALLFED)NashvilleUSA

Personalised recommendations