Abstract
Cyber space, the fifth warfare domain, is benefitting from technological advancements and already accommodating Artificial Intelligence (AI) as its essential part. Enhanced by machine learning capabilities and AI decision making potential, autonomous military systems have been presented as highly efficient new generation weapons ready to replace conventional armaments and human combatants.
Cyber physical weapon systems, ranging from self-targeting missiles to self-learning ground vehicles are boosting covert military offensive capabilities increasing heated discussions and tensions between nations. The AI cyber arms race, escalated by super powers competing for global dominance, is supported by multi-billion-dollar investments into state-of-the-art technologies and leading research institutions. Further uncontrollable AI-based military technology development will expand the threat landscape with self-replicating cyber weapons and autonomous offense.
The chapter reviews the AI impact on autonomy and its major criteria, explores cyber vulnerabilities in autonomous technologies, highlights critical issues of the AI use in AWS, deliberates on incorporation of ethical principles into development of technologies, reveals legal complications and consequences of AI arms race, forecasts future challenges. It also provides some potential crisis scenarios.
Current research arguments that the generated neural networks and machine learning algorithms, being of complex nature, still remain unpredictable, unreliable and even dangerous when fully autonomous. Joint global action, legally bounding regulations and internationally coordinated research are the major solutions to harness and revert the emerging existential danger.
This is a preview of subscription content, access via your institution.
Buying options

Notes
- 1.
Quoted after Cellan-Jones, Rory (2014).
- 2.
Autonomy, Cambridge Dictionary, https://dictionary.cambridge.org/dictionary/english/autonomy
- 3.
Official website at https://www.shield.ai/
- 4.
Neural Network Libraries by Sony, https://nnabla.org
- 5.
Apache MXNet, library for deep learning, http://mxnet.incubator.apache.org/index.html
- 6.
Quoted from Vincent (2018).
- 7.
See (BBC 2016).
- 8.
Aristotle as quoted in Partridge and Hussain (1992).
- 9.
SIPRI databases, https://www.sipri.org/databases
- 10.
Close-In Weapons System.
- 11.
The next paradigm shift: AI-driven Cyber-Attacks, DarkTrace, 2018, https://www.darktrace.com/en/resources/wp-ai-driven-cyber-attacks.pdf
- 12.
MoD’s tests will send satnav haywire so take a road atlas, 6 June 2007, https://www.dailymail.co.uk/news/article-460279/MoDs-tests-send-satnav-haywire-road-atlas.html
- 13.
Exploit – a constructed command or a software designed to take advantage of a flaw in a computer system (vulnerability), typically for malicious purposes, such as accessing system information, establishing remote command line interface, causing denial of service, etc.
- 14.
US military F-18 launch Drone swarm in tests at China Lake, 2017, https://www.youtube.com/watch?v=DjUdVxJH6yI
- 15.
China New Generation Plan (in Chinese), July 2017, at http://www.gov.cn/zhengce/content/2017-07/20/content_5211996.htm
- 16.
Summary of the 2018 Department of Defence Artificial Intelligence Strategy, at https://media.defense.gov/2019/Feb/12/2002088963/-1/-1/1/SUMMARY-OF-DOD-AI-STRATEGY.PDF
- 17.
Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I), 8 June 1977, ICRC database, at https://ihl-databases.icrc.org/applic/ihl/ihl.nsf/Article.xsp?action=openDocument&documentId=FEB84E9C01DDC926C12563CD0051DAF7
- 18.
The Martens clause first appeared in the preamble to the 1899 Hague Convention on the Laws and Customs of War on Land, from the statement done by Fyodor Martens, the Russian delegate at the Hague Peace Conferences of 1899. It states that “Until a more complete code of the laws of war is issued, the High Contracting Parties think it right to declare that in cases not included in the Regulations adopted by them, populations and belligerents remain under the protection and empire of the principles of international law, as they result from the usages established between civilized nations, from the laws of humanity and the requirements of the public conscience”, at https://ihl-databases.icrc.org/applic/ihl/ihl.nsf/Comment.xsp?documentId=42376D4763042448C12581150044672C&action=OpenDocument
- 19.
Meeting of the Group of Governmental Experts on Lethal Autonomous Weapons Systems Geneva, 9 to 13 April 2018, https://www.unog.ch/80256EDD006B8954/(httpAssets)/895931D082ECE219C12582720056F12F/$file/2018_LAWSGeneralExchange_Germany-France.pdf
- 20.
Protocol Additional to the Geneva Conventions of 12 August 1949, and Relating to the Protection of Victims of International Armed Conflict (Additional Protocol I or AP I) (adopted on 8 June 1977, entered into force on 7 December 1978), art 52(2).
- 21.
Asilomar AI Principles, Future of Life Institute, 2017, at https://futureoflife.org/ai-principles
- 22.
United Nations Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons (CCW) regulates weapons which may be deemed to be excessively injurious or to have indiscriminate effects, with Protocols I, II and III, opened for signature 10 April 1981, entered into force 2 December 1983, at https://treaties.un.org/pages/ViewDetails.aspx?src=TREATY&mtdsg_no=XXVI-2&chapter=26&lang=en
- 23.
Top 10 Principles for Ethical Artificial Intelligence, http://www.thefutureworldofwork.org/media/35420/uni_ethical_ai.pdf
- 24.
SyNAPSE Program Develops Advanced Brain-Inspired Chip, 8 July 2014, https://www.darpa.mil/news-events/2014-08-07
References
Abaimov, Stanislav, and Paul Ingram. 2017. Hacking UK Trident: The growing threat, BASIC, https://www.basicint.org/wp-content/uploads/2018/06/HACKING_UK_TRIDENT.pdf.
Anderson, M., and S. Leigh Anderson. 2015. Toward ensuring ethical behavior from autonomous systems: A case-supported principle-based paradigm, https://aaai.org/ocs/index.php/WS/AAAIW15/paper/view/9976/10125.
Arabian Business. 2017. UAE appoints first Minister for Artificial Intelligence. 19 Oct 2017, at https://www.arabianbusiness.com/politics-economics/381648-uae-appoints-first-minister-for-artificial-intelligence.
BBC. 2016. Adobe Voco ‘Photoshop-for-voice’ causes concern, BBC News on 7 November 2016. Last accessed 3 Feb 2019, https://www.bbc.co.uk/news/technology-37899902.
Behrens, J. 2019. FY20 Budget Request: DOD Science and Technology, FYI American Institute of Physics, Bulletin 28. 28 Mar 2019, at https://www.aip.org/fyi/2019/fy20-budget-request-dod-science-and-technology.
Booz Allen Hamilton. 2018. Press Release, U.S. Government & GSA FEDSIM Select Booz Allen to Help Apply Artificial Intelligence, 30 July 2018, last retrieved on 17 Sept 2018 at https://www.boozallen.com/e/media/press-release/booz-allen-selected-to-help-apply-artificial-intelligence.html.
Brandom, R. 2018. Amazon’s facial recognition matched 28 members of Congress to criminal mugshots. At https://www.theverge.com/2018/7/26/17615634/amazon-rekognition-aclu-mug-shot-congress-facial-recognition.
Brundage, Miles, et al. 2018. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. Last retrieved 15 July 2018 at http://img1.wsimg.com/blobby/go/3d82daa4-97fe-4096-9c6b-376b92c619de/downloads/1c6q2kc4v_50335.pdf.
Campaign to Stop Killer Robots. 2018. Country views on killer robots. 22 Nov 2018, https://www.stopkillerrobots.org/wp-content/uploads/2018/11/KRC_CountryViews22Nov2018.pdf.
Cangelosi, A., and M. H. Fischer. 2015. Embodied intelligence, https://www.researchgate.net/publication/283812826_Embodied_Intelligence.
Cassano, Jay. 2018. Pentagon’s artificial intelligence programs get huge boost in defense budget, 15 August 2018. Last retrieved on 18 Aug 2018 at https://www.fastcompany.com/90219751/pentagons-artificial-intelligence-programs-get-huge-boost-in-defense-budget.
Cellan-Jones, Rory. 2014. Stephen Hawking warns artificial intelligence could end mankind, BBC News, 2 December 2014. Last retrieved on 16 Mar 2019 at https://www.bbc.com/news/technology-30290540.
Chair GGE. 2018. Chair’s summary of the discussion on Agenda item 6 (a) 9 and 10 April 2018Agenda item 6 (b) 11 April 2018and 12 April 2018Agenda item 6 (c) 12 April 2018Agenda item 6 (d) 13 April 2018, UNOG, Geneva, https://www.unog.ch/80256EDD006B8954/(httpAssets)/DF486EE2B556C8A6C125827A00488B9E/$file/Summary+of+the+discussions+during+GGE+on+LAWS+April+2018.pdf.
Chen, X., C. Liu, B. Li, K. Lu, and D. Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. Last retrieved 15 July 2018 at https://arxiv.org/pdf/1712.05526.pdf.
Corea, F. 2017. Machine ethics and artificial moral agents, November 2017, https://www.kdnuggets.com/2017/11/machine-ethics-artificial-moral-agents.html.
Cummings, Missy. 2016. see Anderson, Kenneth., Dr. Missy Cummings., Michael A. Newton., Michael Schmitt. (2016). LENS Conference 2016 Autonomous Weapons in the Age of Hybrid War. Duke University School of Law. YouTube. https://www.youtube.com/watch?v=b5mz7Y2FmU4 8:47.
DARPA. 2018. DARPA announces 2 bln USD campaign to develop next wave of AI technologies, FYI. 7 Sept 2018, at https://www.darpa.mil/news-events/2018-09-07.
———. n.d. Mission, defense advanced research projects agency. Last retrieved on 28 Aug 2018 at https://www.darpa.mil/about-us/mission.
Davis, R. 2018. UK unveils new Tempest fighter jet to replace Typhoon, 16 July 2018, at https://www.theguardian.com/uk-news/2018/jul/16/uk-tempest-fighter-jet-typhoon-farnborough-airshow.
Department of Defence (DOD). 2012. The role of autonomy in DOD systems, Task Force Report, Defence Science Board, at https://fas.org/irp/agency/dod/dsb/autonomy.pdf.
Department of Defense (DOD). 2017. Unmanned systems integrated roadmap, 2017–2042, at https://apps.dtic.mil/dtic/tr/fulltext/u2/1059546.pdf.
———. 2018a. Unmanned systems integration roadmap, 2017–2042, at https://apps.dtic.mil/dtic/tr/fulltext/u2/1059546.pdf.
———. 2018b. Memorandum for Chief Management Officer of DOD, 27 June 2018, at https://admin.govexec.com/media/establishment_of_the_joint_artificial_intelligence_center_osd008412-18_r....pdf.
Deputy Secretary of Defense. 2014. Memorandum for Deputy Secretary of Defense, 15 November 2014, https://defenseinnovationmarketplace.dtic.mil/wp-content/uploads/2018/04/DefenseInnovationInitiative.pdf.
Durbin, Ken. 2018. Surge in blended attacks stirs new cyber worries. Last retrieved 19 Feb 2019 from https://www.symantec.com/blogs/expert-perspectives/surge-blended-attacks-stirs-new-cyber-worries.
Dutt, D. 2018. The year of the AI-powered cyberattack, 10 January 2018. Last retrieved on 2 Aug 2018 at https://www.csoonline.com/article/3246196/cyberwarfare/2018-the-year-of-the-ai-powered-cyberattack.html.
Frank, Blair Hanley. 2019. Get ready for AI chips everywhere, AI Weekly 16 February 2018. Last retrieved on 16 Mar 2019 at https://venturebeat.com/2018/02/16/get-ready-for-ai-chips-everywhere/.
Future of Life Institute. 2015. Autonomous weapons: an open letter from AI & robotics researchers, https://futureoflife.org/open-letter-autonomous-weapons/?cn-reloaded=1.
———. 2017. Beneficial AI conference develops ‘Asilomar AI principles’ to guide future AI research, February 2017, http://www.kurzweilai.net/beneficial-ai-conference-develops-asilomar-ai-principles-to-guide-future-ai-research
———. 2018. The risks posed by lethal autonomous weapons, September 2018, https://futureoflife.org/2018/09/04/the-risks-posed-by-lethal-autonomous-weapons/.
———. n.d. Lethal Autonomous Weapons Pledge, Future of Life Institute, https://futureoflife.org/lethal-autonomous-weapons-pledge/.
GGE. 2018. Report of the 2018 session of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, Geneva, 9–13 April 2018 and 27–31 August 2018 at http://undocs.org/en/CCW/GGE.1/2018/3.
Grace, K., et al. 2018. When will AI exceed human performance? Evidence from AI experts. 3 May 2018, at https://arxiv.org/pdf/1705.08807.pdf.
Guterres, António. 2018. Remarks at Web Summit, the United Nations Secretary-General António Guterres, 5 November 2018, https://www.un.org/sg/en/content/sg/speeches/2018-11-05/remarks-web-summit.
H.R.3230 – To combat the spread of disinformation through restrictions on deep-fake video alteration technology, 116th Congress (2019–2020). Accessed 14 June 2019 at https://www.congress.gov/bill/116th-congress/house-bill/3230.
Harbers, Maaike, Marieke M.M. Peeters, and Mark A. Neerincx. 2017. Perceived autonomy of robots: Effects of appearance and context, Chapter 2, https://www.researchgate.net/publication/312412750_Perceived_Autonomy_of_Robots_Effects_of_Appearance_and_Context.
Heyns, Christof. 2013. Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions, 9 April 2013, https://www.ohchr.org/Documents/HRBodies/HRCouncil/RegularSession/Session23/A-HRC-23-47_en.pdf.
Hsu, Jeremy. 2018. Forget killer robots: Autonomous weapons are already online, Undark, 2018. Last retrieved 7 Jan 2019 from https://undark.org/article/killer-robots-autonomous-weapons-online/.
Hurd, W., and R. L. Kelly. 2018. Rise of the machines: Artificial intelligence and its growing impact on U.S. policy, September 2018, at https://www.hsdl.org/?abstract&did=816362.
ICRC. 2014. Report of the ICRC expert meeting on ‘Autonomous weapon systems: Technical, military, legal and humanitarian aspects’, 26–28 March 2014, ICRC, Geneva, 9 May 2014, https://www.icrc.org/en/doc/assets/files/2014/expert-meeting-autonomous-weapons-icrc-report-2014-05-09.pdf.
IEEE. 2017a. Reframing autonomous weapon systems, The IEEE global initiative on ethics of autonomous and intelligent systems, at https://standards.ieee.org/content/dam/ieee-standards/standards/web/documents/other/ead_reframing_autonomous_weapons_v2.pdf.
———. 2017b. The IEEE global initiative on ethics of autonomous and intelligent systems, at https://standards.ieee.org/industry-connections/ec/autonomous-systems.html.
Jha, U.C. 2016. Killer robots: Lethal autonomous weapon systems legal, ethical and moral challenges.
Kott A., et al. 2018. Initial reference architecture of an intelligent autonomous agent in cyber defence, ARL-TR-8337 March 2018.
Kurzweil, R. 2005. The singularity is near: When humans transcend biology. New York: Viking Penguin.
Microsoft. 2017. A digital Geneva convention to protect cyberspace, Microsoft policy papers, https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RW67QH.
———. 2018a. Filling the gaps in international law is essential to making cyberspace a safer place, Microsoft, March 2018, https://www.microsoft.com/en-us/cybersecurity/blog-hub/filling-the-gaps-in-international-law-is-essential-to-making-cyberspace-a-safer-place.
———. 2018b. Digital Peace Now, Microsoft, https://digitalpeace.microsoft.com/.
Muller, V.C., and N. Bostrom. 2013. Future progress in artificial intelligence: A survey of expert opinion, available at https://nickbostrom.com/papers/survey.pdf.
National Science Board. 2018. Science & engineering indicators 2018, online at www.nsf.gov/statistics/2018/nsb20181/assets/nsb20181.pdf.
National Science Foundation (NSF). 2018. NSF and National Science Board Statement on Global Research and Development (R&D) Investments NSB-2018-9, February 2018, online at www.nsf.gov/nsb/news/news_summ.jsp?cntn_id=244465.
Office of Naval Research. n.d. Computational methods for decision making program, available at https://www.onr.navy.mil/en/Science-Technology/Departments/Code-31/All-Programs/311-Mathematics-Computers-Research/computational-methods-resource-optimization.
Office of the Assistant Secretary of Defense for Research and Engineering. 2015. Autonomy community of interest test and evaluation, Verification and Validation Working Group, Technology Investment Strategy 2015–2018, Washington, DC.
Partridge, D., and K.M. Hussain. 1992. Artificial intelligence and business management. Norwood: Ablex Publishing Corporation.
Pecotic, A. 2019. Whoever predicts the future will win the AI arms race, March 2019, at https://foreignpolicy.com/2019/03/05/whoever-predicts-the-future-correctly-will-win-the-ai-arms-race-russia-china-united-states-artificial-intelligence-defense/.
Phrack. 2002. Low cost and portable GPS jammer, Phrack, Issue 60, http://phrack.org/issues/60/13.html#article.
Poole, D., A. Mackworth, and R. Goebel. 1998. Computational intelligence: A logical approach. New York: Oxford University Press.
PWC. 2017. Sizing the prize: PwC’s global artificial intelligence study: Exploiting the AI revolution, https://www.pwc.com/gx/en/issues/analytics/assets/pwc-ai-analysis-sizing-the-prize-report.pdf.
Rohrbeck, Patricia. 2017. Concurrent biological, electromagnetic pulse and cyber-attacks: The ultimate interagency response challenge. Last retrieved on 8 Dec 2018 at http://thesimonscenter.org/wp-content/uploads/2017/05/IAJ-8-2-2017-pg53-61.pdf.
Scharre, P. 2018. Killer robots and autonomous weapons with Paul Scharre, podcast, 1 June 2018. Last retrieved on 2 Aug 2018 at https://www.cfr.org/podcasts/killer-robots-and-autonomous-weapons-paul-scharre.
Schroeder, T.W. 2017. Policies on the employment of lethal autonomous weapon systems in future conflicts, https://www.secnav.navy.mil/innovation/Documents/2017/07/LAWS_Essay.pdf.
Seymour, J., and P. Tully. 2016. Weaponizing data science for social engineering: Automated E2E spear phishing, Twitter. Last retrieved 15 July 2018 at https://www.blackhat.com/docs/us-16/materials/us-16-Seymour-Tully-Weaponizing-Data-Science-For-Social-Engineering-Automated-E2E-Spear-Phishing-On-Twitter-wp.pdf.
Shead, S. 2018. Canada and France create new ‘international panel on AI’. 7 Dec 2018, at https://www.forbes.com/sites/samshead/2018/12/07/canada-and-france-create-new-international-panel-on-ai/#449c4b22ef22.
Simonite, T. 2019. Trump’s plan to keep America first in AI. 11 Feb 2019, at https://www.wired.com/story/trumps-plan-keep-america-first-ai/.
SIPRI. 2017a. Boulanin and Verbruggen, Mapping the development of autonomy in weapon systems, SIPRI Report, 2017, https://www.sipri.org/sites/default/files/Mapping-development-autonomy-in-weapon-systems.pdf.
———. 2017b. SIPRI Yearbook 2017: Armaments, disarmament and international security, summary, SIPRI, 2017, https://www.sipri.org/sites/default/files/2017-09/yb17-summary-eng.pdf.
Soros, G. 2019. Remarks delivered at the World Economic Forum, Davos, Switzerland, January 2019, https://www.georgesoros.com/2019/01/24/remarks-delivered-at-the-world-economic-forum-2/.
Sriskandarajah, D. 2018. Why we need a digital Geneva convention. 23 Apr, 2018, at https://www.diplomaticourier.com/why-we-need-a-digital-geneva-convention/.
Stolton, S. 2018. UN talks: Can the EU stop the charge of the killer robots? 29 Aug 2018, at https://www.euractiv.com/section/digital/news/un-talks-can-the-eu-stop-the-charge-of-the-killer-robots/.
Tarnoff, Ben. 2018. Weaponised AI is coming. Are algorithmic forever wars our future?, The Guardian 11 October 2018, https://www.theguardian.com/commentisfree/2018/oct/11/war-jedi-algorithmic-warfare-us-military.
TrendMicro. 2018. Adapting to the Times: Malware decides Infection, probability with Ransomware or Coinminer, 9 July 2018. Last retrieved on 14 July 2018 at https://www.trendmicro.com/vinfo/se/security/news/cybercrime-and-digital-threats/adapting-to-the-times-malware-decides-infection-profitability-with-ransomware-or-coinminer.
UNDIR. 2018. The weaponization of increasingly autonomous technologies: Artificial intelligence, Report, as adapted from Shane Legg and Marcus Hutter, “A Collection of Definitions of Intelligence”, Technical Report IDSIA-07-07, 15 June 2007, p. 9. Last retrieved on 16 Mar 2019 at http://www.unidir.ch/files/publications/pdfs/the-weaponization-of-increasingly-autonomous-technologies-artificial-intelligence-en-700.pdf.
US Army. 2017. The U.S. Army: Robotic and autonomy systems strategy, U.S. Army Training and Doctrine Command.
Vincent, J. 2018. The Verge Tech report card, AI, at https://www.theverge.com/2018/12/30/18137429/2018-tech-recap-artificial-intelligence-robot-machine-learning-facial-recognition.
Winfield, A., and M. Halverson. 2017. Artificial intelligence and autonomous systems: Why principles matter September 2017 at http://sites.ieee.org/futuredirections/tech-policy-ethics/september-2017/artificial-intelligence-and-autonomous-systems-why-principles-matter/, as quoted from Winfield, A., Written evidence submitted to the UK Parliamentary Select Committee on Science and Technology Inquiry on Robotics and Artificial Intelligence, Discussion Paper, Science and Technology Committee (Commons), Website, 2016. http://eprints.uwe.ac.uk/29428/.
Witschge, L. 2018. Should we be worried about ‘killer robots’? April 2018, https://www.aljazeera.com/indepth/features/worried-killer-robots-180409061422106.html.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Abaimov, S., Martellini, M. (2020). Artificial Intelligence in Autonomous Weapon Systems. In: Martellini, M., Trapp, R. (eds) 21st Century Prometheus. Springer, Cham. https://doi.org/10.1007/978-3-030-28285-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-28285-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28284-4
Online ISBN: 978-3-030-28285-1
eBook Packages: Computer ScienceComputer Science (R0)