Classification of global catastrophic risks connected with artificial intelligence


A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various levels of AI development, namely, (1) before it starts self-improvement, (2) during its takeoff, when it uses various instruments to escape its initial confinement, or (3) after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level.

This is a preview of subscription content, log in to check access.


  1. Alexander S (2016) Ascended economy? Star Slate Codex. Accessed 27 Apr 2018

  2. Anderson M (2017) RethinkX: self-driving electric cars will dominate roads by 2030. In: IEEE Spectrum: technology, engineering and science news. Accessed 17 Jul 2017

  3. (2017) Artificial intelligence startups. Accessed 27 Apr 2018

  4. Armstrong S (2017) Good and safe uses of AI Oracles. ArXiv171105541 Cs

  5. Auerbach D (2014) The Most Terrifying Thought Experiment of All Time. In: Slate. Accessed 27 Apr 2018

  6. Baker BH (2000) The gray matter: the forgotten story of the telephone. Telepress, Kent, WA

    Google Scholar 

  7. Bardi U (2008) The Universal Mining Machine. Accessed 27 Apr 2018

  8. Barrett AM, Baum SD (2017) A model of pathways to artificial superintelligence catastrophe for risk and decision analysis. J Exp Theor Artif Intell 29:397–414

    Google Scholar 

  9. BBC (2017) Cyber-attack: europol says it was unprecedented in scale—BBC News. Accessed 17 Jul 2017

  10. Bender J (2014) Russia may still have an automated nuclear launch system aimed across the northern hemisphere. In: Bus. Insid. Accessed 17 Jul 2017

  11. Blair BG (2011) The logic of accidental nuclear war. Brookings Institution Press, Washington, DC

    Google Scholar 

  12. Boles KS, Kannan K, Gill J et al (2017) Digital-to-biological converter for on-demand production of biologics. Nat Biotechnol 35:672–675 2017

    Google Scholar 

  13. Bostrom N (2002) Existential risks: analyzing human extinction scenarios and related hazards. J Evol Technol 9(1):1–30

    Google Scholar 

  14. Bostrom N (2003a) Astronomical waste: The opportunity cost of delayed technological development. Utilitas 15:308–314

    Google Scholar 

  15. Bostrom N (2003b) Are you living in a computer simulation? Publ Philos Q 53(211):243–255

    Google Scholar 

  16. Bostrom N (2006) What is a singleton. Linguist Philos Investig 5:48–54

    Google Scholar 

  17. Bostrom N (2009) Pascal’s mugging. Analysis 69(3):443–445

    MathSciNet  Google Scholar 

  18. Bostrom N (2011) Infinite ethics. Anal Metaphys 9–59

  19. Bostrom N (2014) Superintelligence. Oxford University Press, Oxford

    Google Scholar 

  20. Bradbury RJ (2001) Matrioshka brains. preprint.

  21. Carrigan RA Jr (2006) Do potential SETI signals need to be decontaminated? Acta Astronaut 58:112–117

    Google Scholar 

  22. Chalmers DJ (2002) Does conceivability entail possibility? In: Gendler T, Hawthorne J (eds) Conceivability possibility. Oxford University Press, New York pp 145–200

    Google Scholar 

  23. Chiew KL, Yong KSC, Tan CL (2018) A survey of phishing attacks: their types, vectors and technical approaches. Expert Syst Appl 106:1–20

    Google Scholar 

  24. Christiano P (2016) Prosaic AI alignment. Accessed 27 Apr 2018

  25. Clavero M, García-Berthou E (2005) Invasive species are a leading cause of animal extinctions. Trends Ecol Evol 20:110

    Google Scholar 

  26. Cole DD, Denkenberger D, Griswold M et al (2016) Feeding everyone if industry is disabled. In: Proceedings of the 6th international disaster and risk conference. Davos, Switzerland

  27. Critch A (2017) Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making (arXiv:1701.01302)

  28. Daniel M (2017) S-risks: why they are the worst existential risks, and how to prevent them (EAG Boston 2017). Accessed 27 Apr 2018

  29. Dennett DC (1978) Why you can’t make a computer that feels pain. Synthese 38:415–456

    Google Scholar 

  30. Ellison H (1967) I have no mouth, and i must scream. Galaxy Publishing Corp, New York

    Google Scholar 

  31. Enserink M (2011) Scientists brace for media storm around controversial flu studies. In: Sciencemag. Accessed 27 Apr 2018

  32. Freitas R (2000) Some limits to global ecophagy by biovorous nanoreplicators, with public policy recommendations. Foresight Institute Technical Report

  33. Future of Life Institute (2016) Accidental nuclear war: a timeline of close calls. Accessed 4 Nov 2017

  34. Futureworld (2013) Airplane “crashes” as hacker gets control. In: Futureworld. Accessed 27 Apr 2018

  35. Gildert S (2011) Why “computronium” is really “unobtanium” IO9. Accessed 27 Apr 2018

  36. Goertzel B (2012) Should humanity build a global ai nanny to delay the singularity until it’s better understood? J Conscious Stud 19(1–2):96–111

    Google Scholar 

  37. Grace K, Salvatier J, Dafoe A et al (2017) When will AI exceed human performance? evidence from AI experts. (arXiv:1705.08807 [cs.AI])

  38. Granoff J (2016) Donald trump is an existential threat to America and the world. Time

  39. Gwern (2016) Why tool AIs want to be agent AIs.

  40. Hanson R (2008) Catastrophe, social collapse, and human extinction. In: Bostrom N, Cirkovic MM (eds) Global catastrophic risks. Oxford University Press, Oxford, p 554

    Google Scholar 

  41. Hanson R (2016) The age of Em: work, love, and life when robots rule the earth. Oxford University Press, Oxford

    Google Scholar 

  42. Hines N (2016) Neural implants could let hackers hijack your brain. In: Inverse. Accessed 17 Jul 2017

  43. Hume D (1739) A treatise of human nature. Oxford: Clarendon Press, London, UK

    Google Scholar 

  44. Hutter M (2000) A theory of universal artificial intelligence based on algorithmic complexity. ArXiv Prepr Cs0004001

  45. Jenkins A (2018) Uber may not be to blame for self-driving car death in Arizona. Fortune, New York

    Google Scholar 

  46. Joy B (2000) Why the future doesn’t need us. Wired, San Francisco, CA

    Google Scholar 

  47. Kahn H (1959) On thermonuclear war. Princeton University Press, Princeton

    Google Scholar 

  48. Kardashev NS (1985) On the inevitability and the possible structures of supercivilizations. Reidel Publishing Co., Dordrecht, pp 497–504

    Google Scholar 

  49. Karpathy A (2015) The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy Blog.

  50. Kushner D (2013) The real story of stuxnet. IEEE Spectr 50:48–53

    Google Scholar 

  51. LaVictoire P, Fallenstein B, Yudkowsky E et al (2014) Program equilibrium in the prisoner’s dilemma via Löb’s theorem. MIRI

  52. LaVictorie P (2015) An Introduction to Löb’s Theorem in MIRI Research. MIRI, San Francisco CA.

    Google Scholar 

  53. Lem S (1961) Return from the stars. Houghton Mifflin Harcourt, Boston, US

    Google Scholar 

  54. Lem S (1963) Summa technologiae. Suhrkamp, Berlin, Germany

    Google Scholar 

  55. Lem S (1973) The Invincible: science fiction. Sidgwick & Jackson, London, UK

    Google Scholar 

  56. Lenat DB, Brown JS (1984) Why AM and EURISKO appear to work. Artif Intell 23:269–294

    Google Scholar 

  57. LoPucki LM (2017) Algorithmic ENTITIES. Social Science Research Network, Rochester

    Google Scholar 

  58. Lubin G (2016) Data reveals the 20 most popular TV shows of 2016. Business Insider

  59. Mennen A (2017) Existential risk from AI without an intelligence explosion.

  60. Menzel C (2017) Actualism. In: Zalta EN (ed) The stanford encyclopedia of philosophy, 2014th edn. Metaphysics Research Lab, Stanford University, Stanford

    Google Scholar 

  61. Meuhlhauser L (2014) How big is the field of artificial intelligence? (initial findings). Accessed 27 Apr 2018

  62. Muehlhauser L (2011) Intelligence explosion FAQ. Accessed 27 Apr 2018

  63. Mullin G (2017) What is the Blue Whale suicide challenge, how many deaths has the game been linked to so far and is it in the UK? TheSun

  64. Oberhaus D (2017) Watch ‘Slaughterbots’, a warning about the future of killer bots. In: Motherboard. Accessed 17 Dec 2017

  65. Omohundro S (2008) The basic AI drives. In: Wang P, Goertzel B, Franklin S (eds) Proceedings of the 2008 conference on Artificial General Intelligence 2008: proceedings of the First AGI Conference. IOS Press Amsterdam, The Netherlands

    Google Scholar 

  66. Orwell G (1948) 1984. Houghton Mifflin Harcourt, Boston, US

    Google Scholar 

  67. Pinker S (2011) The better angels of our nature: The decline of violence in history and its causes. Penguin, London

    Google Scholar 

  68. Reason J (2000) Human error: models and management. BMJ 320:768–770

    Google Scholar 

  69. Russell S (2017) 3 principles for creating safer AI. Accessed 27 Apr 2018

  70. Saito T, Angles J (2013) Hikikomori: adolescence without end. Univesity Of Minnesota Press, Minnesota

    Google Scholar 

  71. Sarma GP, Hay NJ (2016) Mammalian value systems. (arXiv:1607.08289 [cs.AI])

  72. Schneier B (2017) Perspective | The next ransomware attack will be worse than WannaCry. Wash, Post

    Google Scholar 

  73. Shakirov V (2016) Review of state-of-the-arts in artificial intelligence with application to AI safety problem. (ArXiv Prepr ArXiv160504232)

  74. Shulman C (2010) Omohundro’s “basic AI drives” and catastrophic risks. Accessed 27 Apr 2018

  75. Shulman C (2011) Arms races and intelligence explosions. Singularity Hypotheses. Springer, New York

    Google Scholar 

  76. Sotala K (2016) Decisive strategic advantage without a hard takeoff. Accessed 27 Apr 2018

  77. Sotala K (2017) Disjunctive AI scenarios: Individual or collective takeoff? Accessed 27 Apr 2018

  78. Sotala K, Yampolskiy R (2014) Responses to catastrophic AGI risk: a survey. Phys Scr 90:018001

    Google Scholar 

  79. Srugatsky N, Strugatsky B (1985) The time wanderers. Richardson & Steirman, New York, US

    Google Scholar 

  80. Strugatsky A, Strugatsky B (1976) The final circle of paradise, Translated by Leonid Renen. DAW, New York

    Google Scholar 

  81. Taylor A (2017) Flying around the world in a solar powered plane—the Atlantic

  82. The Telegraph (2009) Russian spacecraft landed on moon hours before Americans. The telegraph. Accessed 27 Apr 2018

  83. Torres P (2014) Why running simulations may mean the end is near. Accessed 27 Apr 2018

  84. Torres P (2016) Problems with defining an existential risk. IEET. Accessed 27 Apr 2018

  85. Turchin A (2018) The risks connected with possibility of finding alien AI code during SETI. Rev J Br Interplanet Soc. Manuscript,

  86. Turchin A, Denkenberger D (2017) Levels of self-improvement. Manuscript, University of Louisville, TN

  87. Turchin A, Denkenberger D (2018a) Military AI as convergent goal of the self-improving AI. In: Yampolskiy R (ed) Artificial intelligence safety and security. CRC Press, Baca Raton

    Google Scholar 

  88. Turchin A, Denkenberger D (2018b) Could slaughterbots wipe out humanity? Assessment of the global catastrophic risk posed by autonomous weapons. Manuscript

  89. Turchin A, Green B, Denkenberger D (2017) multiple simultaneous pandemics as most dangerous global catastrophic risk connected with bioweapons and synthetic biology. Rev Health Secur

  90. Turing AM (1937) On computable numbers, with an application to the Entscheidungsproblem. Proc Lond Math Soc 2:230–265

    MathSciNet  MATH  Google Scholar 

  91. Velicovich B (2017) I could kill you with a consumer drone. Defense one, Washington, DC

    Google Scholar 

  92. Watkins J (2016) “Shut up and dance”—“Black mirror” series

  93. Wei D (2013) Outside view(s) and MIRI’s FAI endgame. Accessed 27 Apr 2018

  94. Wootson J (2017) Elon Musk doesn’t think we’re prepared to face humanity’s biggest threat: artificial intelligence. Wash, Post

    Google Scholar 

  95. Yampolskiy R (2014) Utility function security in artificially intelligent agents. J Exp Theor Artif Intell JETAI 373–389.

    Google Scholar 

  96. Yampolskiy R (2015a) Artificial superintelligence: a futuristic approach. CRC Press, Boca Raton

    Google Scholar 

  97. Yampolskiy R (2015b) Taxonomy of pathways to dangerous AI. (ArXiv Prepr ArXiv151103246)

  98. Yampolskiy R, Spellchecker M (2016) artificial intelligence safety and cybersecurity: a timeline of AI failures. (ArXiv Prepr ArXiv161007997)

  99. Yudkowsky E (2001) Creating friendly AI 1.0: the analysis and design of benevolent goal architectures. MIRI, San Francisco, CA, pp 1–282

    Google Scholar 

  100. Yudkowsky E (2002) The AI-Box Experiment. Accessed 27 Apr 2018

  101. Yudkowsky E (2003) HUMOR: friendly AI critical failure table. Accessed 27 Apr 2018

  102. Yudkowsky E (2004) Coherent extrapolated volition. Accessed 27 Apr 2018

  103. Yudkowsky E (2008) Artificial intelligence as a positive and negative factor in global risk, in global catastrophic risks. Oxford University Press, Oxford

    Google Scholar 

  104. Yudkowsky E (2015) From AI to zombies. MIRI, San Francisco, CA

    Google Scholar 

  105. Yudkowsky E (2017) Comment on paper clip maximiser scenario. Accessed 27 Apr 2018

  106. Yudkowsky E, Hanson R (2008) The Hanson-Yudkowsky AI-foom debate. In: MIRI Technical report

  107. Yudkowsky E, Herreshoff M (2013) Tiling agents for self-modifying AI, and the Löbian obstacle. Early Draft MIRI

Download references


We would like to thank Roman Yampolskiy and Seth Baum for their interesting ideas in this article. This article represents views of the authors and does not necessarily represent the views of the Global Catastrophic Risk Institute or the Alliance to Feed the Earth in Disasters. No external sources of funding were used for this work.

Author information



Corresponding author

Correspondence to Alexey Turchin.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Turchin, A., Denkenberger, D. Classification of global catastrophic risks connected with artificial intelligence. AI & Soc 35, 147–163 (2020).

Download citation


  • Artificial intelligence
  • Global risks
  • Military drones
  • Superintelligence
  • Existential risk