This paper presents a simple model of an AI (artificial intelligence) arms race, where several development teams race to build the first AI. Under the assumption that the first AI will be very powerful and transformative, each team is incentivised to finish first—by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI disaster, especially if risk-taking is more important than skill in developing the AI. Surprisingly, information also increases the risks: the more teams know about each others’ capabilities (and about their own), the more the danger increases. Should these results persist in more realistic models and analysis, it points the way to methods of increasing the chance of the safe development of AI.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Though high uncertainties do not imply safety.
And only the winning team—if another team gets a disastrous AI first by taking lower precautions, they will ‘won’ the race to build the first AI.
Of course, the model can be refined in various ways. One could make capacity information uncertain and fuzzy, one could have different levels of enmity between different teams, one could incorporate uncertainty about the safety levels and the ultimate outcomes, and so on. Or one could have a dynamic process to determine the outcome, rather than rushing straight to the Nash equilibrium. But the simple model is enough to gain useful insights.
It may seem unusual for teams to not know their own capabilities in the real world. However, this is close to the situation we find ourselves with current AI research: people and organisations have a pretty clear idea of what resources and knowledge they have, but do not know how hard AI is or what routes are most likely to lead there. They are thus effectively ignorant of their own AI-building capabilities.
If makes no sense that a team with higher capability would have a lower chance of winning (if so, they would voluntarily destroy part of their capability).
Such secrecy can interfere with trust building, though, making it hard to reach agreements between teams if such agreement is needed.
This is because only the teams with low capability take risks in cases of private information, and the more teams there are, the less likely it is that the winner will be low capability.
Armstrong S (2013) General purpose intelligence: arguing the orthogonality thesis. Anal Metaphys 12:68–84
Armstrong S, Pamlin D (2015) 12 Risks that threaten human civilisation. Global challenges foundation, http://globalchallenges.org/publications/globalrisks/about-the-project/
Armstrong S, Sotala K (2012) How we’re predicting ai-or failing to. In: Romportl J, Ircing P, Zackova E, Polak M, Schuster R (eds) Beyond AI: artificial dreams. University of West Bohemia, Pilsen, pp 52–75
Armstrong S, Sandberg A, Bostrom N (2012) Thinking inside the box: controlling and using an oracle ai. Minds Mach 22:299–324
Armstrong S, Sotala K, hÉigeartaigh SO (2014) The errors, insights and lessons of famous ai predictions—and what they mean for the future. In: Proceedings of the 2012 Oxford Winter intelligence conference
Bostrom N (2011) Information hazards: a typology of potential harms from knowledge. Rev Contemp Philos 10:44–79
Bostrom N (2012) The superintelligent will: motivation and instrumental rationality in advanced artificial agents. Minds Mach 22:71–85
Bostrom N (2013) Existential risk prevention as global priority. Glob Policy 4:15–31
Bostrom N (2014) Superintelligence: paths, dangers, strategies. Oxford University Press, Oxford
Bostrom N, Sandberg A, Douglas T (2013) The unilateralist’s curse: the case for a principle of conformity. http://www.nickbostrom.com/papers/unilateralist.pdf
Chalmers D (2010) The singularity: a philosophical analysis. J Conscious Stud 17:7–65
Good I (1965) Speculations concerning the first ultraintelligent machine. Adv Comput 6:31–83
Hardin G (1968) The tragedy of the commons. Science 162:1243–1248
Henkel RD, Miller T, Weyant RS (2012) Monitoring select agent theft, loss and release reports in the United States–2004–2010. Appl Biosaf 17:171–180
Heymann DL, Aylward RB, Wolff C (2004) Dangerous pathogens in the laboratory: from smallpox to today’s sars setbacks and tomorrow’s polio-free world. Lancet 363:1566–1568
Kahn H (1960) On thermonuclear war. Transaction publishers, Piscataway
Muehlhauser L, Salamon A (2012) Intelligence explosion: evidence and import. In: Eden A, Søraker J, Morr J, Steinhart E (eds) The singularity hypothesis: a scientific and philosophical assessment. Springer, Berlin
Omohundro SM (2008) The basic AI drives. Front Artif Intell Appl 171:483–492
Sandberg A, Bostrom N (2011) Machine intelligence survey. Technical report, future of humanity institute, Oxford University #2011-1:1–12
Yudkowsky E (2008) Artificial intelligence as a positive and negative factor in global risk. In: Bostrom N, Ćirković MM (eds) Global catastrophic risks. Oxford University Press, New York, pp 308–345
We are grateful to Anders Sandberg and Seán S. ÓhÉigeartaigh for helpful discussions.
About this article
Cite this article
Armstrong, S., Bostrom, N. & Shulman, C. Racing to the precipice: a model of artificial intelligence development. AI & Soc 31, 201–206 (2016). https://doi.org/10.1007/s00146-015-0590-y
- Artificial intelligence
- Arms race
- Coordination problem