Skip to main content
Log in

Racing to the precipice: a model of artificial intelligence development

  • Open Forum
  • Published:
AI & SOCIETY Aims and scope Submit manuscript


This paper presents a simple model of an AI (artificial intelligence) arms race, where several development teams race to build the first AI. Under the assumption that the first AI will be very powerful and transformative, each team is incentivised to finish first—by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI disaster, especially if risk-taking is more important than skill in developing the AI. Surprisingly, information also increases the risks: the more teams know about each others’ capabilities (and about their own), the more the danger increases. Should these results persist in more realistic models and analysis, it points the way to methods of increasing the chance of the safe development of AI.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others


  1. Though high uncertainties do not imply safety.

  2. Some examples of bad values could be ‘prevent human suffering’ via killing people off and ‘make people happy’ via wireheading(Yudkowsky 2008; Bostrom 2014).

  3. And only the winning team—if another team gets a disastrous AI first by taking lower precautions, they will ‘won’ the race to build the first AI.

  4. Of course, the model can be refined in various ways. One could make capacity information uncertain and fuzzy, one could have different levels of enmity between different teams, one could incorporate uncertainty about the safety levels and the ultimate outcomes, and so on. Or one could have a dynamic process to determine the outcome, rather than rushing straight to the Nash equilibrium. But the simple model is enough to gain useful insights.

  5. It may seem unusual for teams to not know their own capabilities in the real world. However, this is close to the situation we find ourselves with current AI research: people and organisations have a pretty clear idea of what resources and knowledge they have, but do not know how hard AI is or what routes are most likely to lead there. They are thus effectively ignorant of their own AI-building capabilities.

  6. If makes no sense that a team with higher capability would have a lower chance of winning (if so, they would voluntarily destroy part of their capability).

  7. Such secrecy can interfere with trust building, though, making it hard to reach agreements between teams if such agreement is needed.

  8. This is because only the teams with low capability take risks in cases of private information, and the more teams there are, the less likely it is that the winner will be low capability.


  • Armstrong S (2013) General purpose intelligence: arguing the orthogonality thesis. Anal Metaphys 12:68–84

    Google Scholar 

  • Armstrong S, Pamlin D (2015) 12 Risks that threaten human civilisation. Global challenges foundation,

  • Armstrong S, Sotala K (2012) How we’re predicting ai-or failing to. In: Romportl J, Ircing P, Zackova E, Polak M, Schuster R (eds) Beyond AI: artificial dreams. University of West Bohemia, Pilsen, pp 52–75

    Google Scholar 

  • Armstrong S, Sandberg A, Bostrom N (2012) Thinking inside the box: controlling and using an oracle ai. Minds Mach 22:299–324

    Article  Google Scholar 

  • Armstrong S, Sotala K, hÉigeartaigh SO (2014) The errors, insights and lessons of famous ai predictions—and what they mean for the future. In: Proceedings of the 2012 Oxford Winter intelligence conference

  • Bostrom N (2011) Information hazards: a typology of potential harms from knowledge. Rev Contemp Philos 10:44–79

    Google Scholar 

  • Bostrom N (2012) The superintelligent will: motivation and instrumental rationality in advanced artificial agents. Minds Mach 22:71–85

    Article  Google Scholar 

  • Bostrom N (2013) Existential risk prevention as global priority. Glob Policy 4:15–31

    Article  Google Scholar 

  • Bostrom N (2014) Superintelligence: paths, dangers, strategies. Oxford University Press, Oxford

    Google Scholar 

  • Bostrom N, Sandberg A, Douglas T (2013) The unilateralist’s curse: the case for a principle of conformity.

  • Chalmers D (2010) The singularity: a philosophical analysis. J Conscious Stud 17:7–65

    Google Scholar 

  • Good I (1965) Speculations concerning the first ultraintelligent machine. Adv Comput 6:31–83

    Article  Google Scholar 

  • Hardin G (1968) The tragedy of the commons. Science 162:1243–1248

    Article  Google Scholar 

  • Henkel RD, Miller T, Weyant RS (2012) Monitoring select agent theft, loss and release reports in the United States–2004–2010. Appl Biosaf 17:171–180

    Article  Google Scholar 

  • Heymann DL, Aylward RB, Wolff C (2004) Dangerous pathogens in the laboratory: from smallpox to today’s sars setbacks and tomorrow’s polio-free world. Lancet 363:1566–1568

    Article  Google Scholar 

  • Kahn H (1960) On thermonuclear war. Transaction publishers, Piscataway

    Google Scholar 

  • Muehlhauser L, Salamon A (2012) Intelligence explosion: evidence and import. In: Eden A, Søraker J, Morr J, Steinhart E (eds) The singularity hypothesis: a scientific and philosophical assessment. Springer, Berlin

    Google Scholar 

  • Omohundro SM (2008) The basic AI drives. Front Artif Intell Appl 171:483–492

    Google Scholar 

  • Sandberg A, Bostrom N (2011) Machine intelligence survey. Technical report, future of humanity institute, Oxford University #2011-1:1–12

  • Yudkowsky E (2008) Artificial intelligence as a positive and negative factor in global risk. In: Bostrom N, Ćirković MM (eds) Global catastrophic risks. Oxford University Press, New York, pp 308–345

    Google Scholar 

Download references


We are grateful to Anders Sandberg and Seán S. ÓhÉigeartaigh for helpful discussions.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Stuart Armstrong.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Armstrong, S., Bostrom, N. & Shulman, C. Racing to the precipice: a model of artificial intelligence development. AI & Soc 31, 201–206 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: