Abstract
AI-based, autonomous weapon systems (AWS) have the potential of weapons of mass destruction and thereby massively add to the intensifying dialectic of fear between ground and space and the pervasive mass human vulnerability of being tracked and targeted from above. Nevertheless, the dangerous effects of the proliferation of AWS have not been and still are not widely acknowledged. On the one hand, the capabilities and effects of AWS are downplayed by the military and the arms industry staging these systems as precise and clean. Recently, it is also argued that they can be built on the basis of a ‘responsible’ or ‘trustworthy’ artificial intelligence (AI). On the other hand, inadequate sociotechnical imaginaries of AI as a conscious, evil super-intelligence circulated by Hollywood blockbuster films such as 'Terminator' or 'Ex Machina' dominate the public discourse. Their massive overstatement of the power of the technology and also their focus on often irrelevant imaginaries such as the ‘Terminator’ hinders a realistic understanding of the AI’s capabilities. Against this background, arms control advocates develop new imaginaries to show the loss of ‘meaningful human control’ (Sharkey 2016) and its problematic consequences. In October 2023, the deployment of autonomous military in the battlefield has already been officially confirmed by an Ukrainian drone company (Hambling 2023).
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Negotiations over a ban on lethal AWS have been ongoing at the Convention on Certain Conventional Weapons (CCW) in Geneva since 2014. In parallel, the problematic impact of AWS, as well as tele-operated drones, has been critically debated in academia (Gregory 2011, Bhuta et al. 2016, Suchman and Weber 2016, Weber 2016) as well as by non-governmental and investigative journalist organizations, i.e., such as the Campaign to Stop Killer Robots, Code Pink, the Bureau of Investigative Journalism, the International Committee for Robot Arms Control (ICRAC). The massive violations of human rights by tele-operated drones were documented and discussed while the possible severe humanitarian consequences of the deployment of AWS were outlined.
Nevertheless, the efforts of human rights and arms control advocates have had little effect in the public and political arena so far. Against all warnings of the devastating consequences of these systems for civilians - while also ignoring the problem that these systems serve as a trigger for a new global arms race -, drones are staged as ‘precise and clean’ and a remedy to save the lives of one’s own soldiers, by the military, the defense industry, as well as politicians. Against this background, AWS have at the same time been staged for many years as a problem that will only become reality in the far future or that they will always be controllable through a human on the loop or through ‘responsible AI’ (Scharre 2018).
The interest in AWS is rising in many nations as loitering munition (so-called kamikaze drones) is increasingly discussed in the media as the decisive game changer for military advantage, for example already in the Nagorno-Karabakh war (Deutsche Welle 2021) and today especially in the Russian-Ukraine war (Hambling 2023). And again, there is little debate about the tremendous consequences of these weapons for civilians as AWS turn into a permanent threat from above: ‘Some next-generation military drones rely on artificial intelligence to circle over an area, pick out enemy units and destroy them. In the coming years, drone technology will improve, and the cost of drones will decline. As they do, the frightening truth is that troops and civilians in future conflicts will find fewer and fewer places to hide from the gaze of both man and machine’ (Kingsbury 2022). The recently started Israel-Gaza war is a terrifying example for this development (Davies et al. 2023).
Against this background, several legal, social science and humanities scholars as well as journalists and activists claim that a ‘new human right to protect the freedom to live without physical or psychological threat from above’ (Grief et al. 2018) are required. This proposed new human right has been developed in the Airspace Tribunal hearings in London, Sydney, Toronto and Berlin in the last years.
Contested imaginaries of AI
According to some AI experts, an enormous obstacle to a realistic debate about the potentials of AWS is the sociotechnical resp. pop-cultural imaginaries of AI shaped by Hollywood blockbuster films such as the Terminator series, ‘Ex Machine’, or ‘I, Robot’. These films often stage autonomous AI dramatically as a conscious, evil super-intelligence striving for the erasure of the human race. While the power of the technology is massively overstated, it is at the same time neutralized through this overstatement: The popular discourse revolves around the question of whether AI can gain consciousness, while the concrete effects of applied AI, such as the loss of meaningful human control (Sharkey 2016) and the reconfiguration of human–machine relations (Suchman and Weber 2016), have been mostly overlooked. ‘We have witnessed high-level defense officials dismissing the risk on the grounds that their “experts” do not believe that the “Skynet thing” is likely to happen. Skynet, of course, is the fictional command and control system in the Terminator movies that turns against humanity. The risk of the “Skynet thing” occurring is completely unconnected to the risk of humans using autonomous weapons as WMDs or to any of the other risks […]. If even senior defence officials with responsibility for autonomous weapons programs fail to understand the core issues, then we cannot expect the general public and their elected representatives to make appropriate decisions’ (Russell et al. 2018).
‘Enough to kill half a city’
Arms control advocates—from peace researchers to computer scientists—aim to foster a more realistic imaginary of AWS and to illustrate their deadly potential. The Future of Life Institute and well-known AI expert Stuart Russell together with a professional film team developed short YouTube videos to make the consequences explicit for a broader audience. The first video called ‘Slaughterbots’ went viral after its release in 2017. It had received more than two million clicks within a few days, even though it was not an SF trailer but a science communication video. The Slaughterbots video starts with a typical CEO presentation in which the protagonist demonstrates the capabilities of the emergent drone swarms, released in hundreds or thousands from an aeroplane, which allow according to the CEO an ‘airstrike of surgical precision … A 25-million-dollar order now buys this … Enough to kill half a city, the bad half’, because it ‘allows you to separate the good guys from the bad’ (Slaughterbots 2017). The drones are equipped with face recognition software to follow and kill selected targets—according to their social media profiles, for example. With this new weapons system, the CEO claims, ‘nuclear is obsolete’ (ibid.). The rest of the video develops two main scenarios in which critical members of parliament and hundreds of politically engaged students are lethally attacked by drone swarms. Stuart Russell warns of the problems and effects of autonomous weapons: ‘What we were trying to show was the property of autonomous weapons to turn into weapons of mass destruction automatically because you can launch as many as you want’ (ibid.). The video impressively sketches the potential for mass destruction of AWS—also in the civilian context. Russell makes clear that the dangerous capabilities of AWS shown in the film are not decades away (as often claimed by some countries at the CCW talks in Geneva), but the ‘results of integrating and militarizing technologies that we already have’ (ibid.)—and that this development needs to be stopped: ‘Allowing machines to choose to kill humans will be devastating to our security and freedom. We have an opportunity to prevent the future you just saw, but the window to act is closing fast’ (ibid.). In the meantime, the first autonomous drones seem to operate. In 2020, an UN report stated that the first autonomous drones have been used in Libya (United Nations, Security Council Report on Second Libyan Civil War 2020, 17/548) but this statement was never confirmed. In 2023, the Ukrainian drone company Saker proudly announced the use of autonomous drones (Hambling 2023).
Autonomy
One of the key issues of understanding AWS is the complex and multiple meanings of autonomy. While in the humanities and social sciences as well as everyday life, autonomy is associated with a free and self-aware subject which acts self-determinedly and consciously. Even though this Kantian concept has been challenged by well-known theorists from Karl Marx to Judith Butler, it still predominates in realms such as ethics, law, economics and everyday life. The concept of autonomy used in AI and robotics has a very different meaning: It follows a cybernetic concept of purposeful behavior in the sense of a pragmatic physiological automated mechanism, like the target-seeking mechanism of a torpedo. Today’s control mechanisms in AI systems are much more sophisticated than traditional servomechanisms but nevertheless AWS do neither follow their ‘own’ rules nor are they capable of decision-making in a wider, (self-)reflective sense. They are determined by norms, values and categories which were programmed into the software by computer scientists—and although the complexity of software layers might lead to unpredictable effects, these are not intentional (Suchman/Weber 2016). The Slaughterbots drones can find and follow targets. For example, they use social media profiles to find and follow the people to be killed. But this behaviour is preprogrammed. This sophisticated entanglement of autonomous and preprogrammed behavior in autonomous systems makes it so difficult to understand the challenges they pose.
The arms control imaginary: WMDs
In the Slaughterbots video, it becomes obvious that autonomous drone swarms are not self-determined and self-conscious intelligent ‘organisms.’ The ‘Slaughterbots’ are programmed to select their targets via data analytics according to pre-given criteria: For example, identifying, searching and targeting leftist students engaged in an anti-corruption NGO via their social media profiles. The bots seek their targets using facial recognition and to kill them with explosives. The Slaughterbots may show coordinated, flexible behavior to perform their tasks (avoiding obstacles, following humans, etc.), but these swarms are neither conscious nor capable of setting their own agendas and develop their own goals. The arms control imaginary strives to show the decisive difference between a Hollywood imaginary of the self-conscious, intelligent, autonomous AI and a more realistic Slaughterbots imaginary of AI as a collection of smart software programs.
The imaginary the Slaughterbots’ video emphasizes that today’s AI makes it possible to automate sophisticated and sensible tasks that are normally performed by humans. These software programs are not intelligent in themselves. Nevertheless, adaptive, coordinated drones as well as drone swarms can easily be turned into WMDs.
The arms control advocates’ Slaughterbots video is in my view an important step toward the development of a new AI imaginary that is not build on the old trope of the evil, almighty wrongdoer, but which makes the eminent questions of arms control of AI-based systems and the dimension of lethal autonomous weapon systems as weapons of mass destruction visible. Today autonomous drones are already in operation though many activists are working towards the ban of such systems. This ban is desperately needed as an important contribution to ending the fear of being tracked and targeted from above as demanded by the Airspace Tribunal.
Notes
-
1.
This essay partially relies on an earlier paper: ‘Artificial Intelligence and the Sociotechnical Imaginary: On Skynet, Self-Healing Swarms and Slaughterbots.’ In: Kathrin Maurer, Andreas Immanuel Graae (Eds.): Drone Imaginaries and the Power of Vision. Manchester: Manchester University Press 2021
-
2.
Autonomous weapons are defined as systems ‘that, once activated can track, identify, and attack targets with violent force without further human interaction’ (Sharkey 2016, 3)
-
3.
For the concept of the imaginary see Jasanoff and Kim (2009, 2015), Mager and Katzenbach (2021), McNeil et al. (2017)
-
4.
Slaughterbots, directed by S. Sugg, written by M. Wood, YouTube (2017), last accessed 10.10.2022, www.youtube.com.
References
Bhuta, N., S. Beck, R. Geiss, H.-Y. Liu, and C. Kress, eds. 2016. Autonomous Weapon Systems: Law, Ethics, Policy. Cambridge: Cambridge University Press.
Davies, Harry., Bethan McKernan, and Dan Sabbagh. 2023. ‘The Gospel’: how Israel uses AI to select bombing targets in Gaza. The Guardian. https://www.theguardian.com/world/2023/dec/01/the-gospel-how-israel-uses-aito-select-bombing-targets. Accessed 20 Dec 2023.
Deutsche Welle. 2021. The Future of Modern Warfare: How Technology is Transforming Conflict. https://www.youtube.com/watch?v=TmlBkW6ANsQ. Accessed 10 Oct 2022.
Graae, A.I., and K. Maurer, eds. 2020. Drone Imaginaries and the Power of Vision. Manchester: Manchester University Press.
Gregory, D. 2011. From a View to a Kill: Drones and Late Modern War. Theory, Culture & Society 28 (7–8): 188–215.
Grief, N., S. Illingworth, A. Hoskins, and C. Conway. 2018. The Airspace Tribunal: Towards a New Human Right to Protect the Freedom to Exist Without a Physical or Psychological Threat from Above. European Human Rights Law Review. 3: 201–207.
Hambling, D. 2023 Ukraine’s AI Drones Seek And Attack Russian Forces Without Human Oversight. https://www.forbes.com/sites/davidhambling/2023/10/17/ukraines-ai-drones-seek-and-attack-russian-forces-without-human-oversight/. Accessed 16 Nov 2023.
Jasanoff, S., and S.-H. Kim. 2009. Containing the Atom: Sociotechnical Imaginaries and Nuclear Power in the United States and South Korea. Minerva 47 (2): 119–146.
Jasanoff, S., and S.-H. Kim, eds. 2015. Dreamscapes of Modernity: Sociotechnical Imaginaries and the Fabrication of Power, 1–33. Chicago: University of Chicago Press.
Kingsbury, A. 2022. No One Can Hide from this Weapon in the War in Ukraine. New York Times. https://www.nytimes.com/2022/05/30/opinion/drones-ukraine-war.html?searchResultPosition=8. Accessed 30 May 2022.
Mager, A., and C. Katzenbach. 2021. Future Imaginaries in the Making and Governing of Digital Technology: Multiple, Contested, Commodified. New Media & Society 23 (2): 223–236.
McNeil, M., M. Arribas-Ayllon, J. Haran, A. Mackenzie, and R. Tutton. 2017. Conceptualizing Imaginaries of Science, Technology, and Society. In The Handbook of Science and Technology Studies, 4th ed., ed. U. Felt, R. Fouché, C.A. Miller, and L. Smith-Doerr, 435–463. Cambridge: MIT Press.
Russell, S., Aguirre, A., Conn, A., and Tegmark, M. (2018). Why You Should Fear “Slaughterbots”: A Response. IEEE Spectrum. https://spectrum.ieee.org. Accessed 10 Oct 2022.
Scharre, P. 2018. Army of None. Autonomous Weapons and the Future of Warfare. New York: W.W. Norton & Company.
Sharkey, Noel. 2016. Staying in the Loop: Human Supervisory Control of Weapons. In Autonomous Weapons Systems: Law, Ethics, Policy, ed. Nehal Bhuta, et al., 23–38. Cambridge: Cambridge University Press.
Slaughterbots. 2017. Directed by S. Sugg, Written by M. Wood, YouTube. www.youtube.com. Accessed 10 Oct 2022.
Suchman, L., and J. Weber. 2016. Human-Machine Autonomies. In Autonomous Weapon Systems: Law, Ethics, Policy, ed. N. Bhuta, S. Beck, R. Geiss, H.-Y. Liu, and C. Kress, 75–102. Cambridge: Cambridge University Press.
United Nations, Security Council 2021. Letter dated 8 March 2021 from the Panel of Experts on Libya established pursuant to resolution 1973 (2011) addressed to the President of the Security Council.
Weber, J. 2016. Keep Adding: Kill Lists, Drone Warfare and the Politics of Databases. In: M. DeGoede, A. Leander and G. Sullivan (eds), ‘The Politics of the List: Law, Security, Technology’, Special Issue, Environment and Planning D: Society and Space, vol. 34, no. 1, eds. M. DeGoede, A. Leander and G. Sullivan, 107–125.
Funding
Open Access funding enabled and organized by Projekt DEAL.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Weber, J. Autonomous drone swarms and the contested imaginaries of artificial intelligence. Digi War 5, 146–149 (2024). https://doi.org/10.1057/s42984-023-00076-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1057/s42984-023-00076-7