Autonomous Machines, Moral Judgment, and Acting for the Right Reasons
KeywordsKilling War Autonomous Autonomous weapons Just war theory Right reasons Moral judgment Driverless cars Responsibility Artificial intelligence
Modern weapons of war have undergone precipitous technological change over the past generation and the future portends even greater advances. Of particular interest are so-called ‘autonomous weapon systems’ (henceforth, AWS), that will someday purportedly have the ability to make life and death targeting decisions ‘on their own.’ Many have strong moral intuitions against such weapons, and public concern over AWS is growing. A coalition of several non-governmental organizations, for example, has raised the alarm through their highly publicized ‘Campaign to Stop Killer Robots’ in an effort to enact an international ban on fully autonomous weapons.1 Despite the strong and widespread sentiments against such weapons, however, proffered philosophical arguments against AWS are often found lacking in substance.
We propose that the prevalent moral aversion to AWS is supported by a pair of compelling objections. First, we argue that even a sophisticated robot is not the kind of thing that is capable of replicating human moral judgment. This conclusion follows if human moral judgment is not codifiable, i.e., it cannot be captured by a list of rules. Moral judgment requires either the ability to engage in wide reflective equilibrium, the ability to perceive certain facts as moral considerations, moral imagination, or the ability to have moral experiences with a particular phenomenological character. Robots cannot in principle possess these abilities, so robots cannot in principle replicate human moral judgment. If robots cannot in principle replicate human moral judgment then it is morally problematic to deploy AWS with that aim in mind. Second, we then argue that even if it is possible for a sufficiently sophisticated robot to make ‘moral decisions’ that are extensionally indistinguishable from (or better than) human moral decisions, these ‘decisions’ could not be made for the right reasons. This means that the ‘moral decisions’ made by AWS are bound to be morally deficient in at least one respect even if they are extensionally indistinguishable from human ones. Our objections to AWS support the prevalent aversion to the employment of AWS in war. They also enjoy several significant advantages over the most common objections to AWS in the literature.
The most well-known objection to autonomous weapons systems (AWS) is that their deployment would result in what are referred to as ‘responsibility gaps.’2 Like other objections to AWS, this is a contingent problem that could in theory be solved by the perfection of artificial intelligence (henceforth, AI). The objection to AWS we defend below is not contingent on AWS making mistakes. Further, a point that has not been fully appreciated is that many of these objections to AWS would rule out technologies which are intuitively less repugnant, or even attractive, such as driverless cars. We should prefer an objection to AWS that could distinguish between AWS and other autonomous decision-making technologies. Below we show that our objection could justify a moral distinction between weaponized and non-weaponized autonomous technologies. In our closing remarks, we propose that if AWS reached a level of sophistication that made them better than humans at making moral judgments, this would alleviate worries about their effectiveness in war, but it would ultimately raise much deeper concerns about the centrality of moral judgment to a meaningful human life.
2 Previous Objections to AWS
Let us imagine a future state of highly advanced so-called ‘autonomous’ weapons. These are weapons which are able to ‘make decisions’ via an artificial intelligence regarding the targeting and killing of human beings in some sense that is ‘on their own’ and separate from human agency. In Robert Sparrow’s terms, to call an agent autonomous is to say that “their actions originate in them and reflect their ends. Furthermore, in a fully autonomous agent, these ends are ends that they have themselves, in some sense, chosen” (2007: 65). More recently, Sparrow has offered the following, more metaphysically neutral, definition of autonomous agency: “an ‘autonomous’ weapon is capable of being tasked with identifying possible targets and choosing which to attack, without human oversight, and that is sufficiently complex such that, even when it is functioning perfectly, there remains some uncertainty about which targets it will attack and why” (2013: 4).3 Many weapons systems in use today demonstrate some degree of autonomy, in the sense that they can choose to engage targets, choose a path to their target, or even choose what munitions to ultimately detonate on impact (Sparrow 2007: 63–64).4 Another way to capture the kind of technology we are here envisioning is on Tjerk de Greef’s capabilities scale (De Greef et al. 2010). We are focused on those kinds of weapons which would be classified as having a “High” level of autonomy on De Greef’s scale. That is, at the most extreme end of the spectrum (what De Greef calls “Level 10”), we are imagining weapons that can act in such a way that “the computer decides everything, acts autonomously, ignoring the human.”5 Many roboticists view an artificial intelligence with this level of autonomy as decades away; others believe this level of autonomy is ultimately impossible. If weapons with this level of autonomy are accessible at all, however, we are confident they will eventually be widely used in battle.,67
Waging war requires that we are able to justly hold someone morally responsible for the deaths of enemy combatants that we cause.
Neither the programmer of AWS nor its commanding officer could justly be held morally responsible for the deaths of enemy combatants caused by AWS.
We could not justly hold AWS itself morally responsible for its actions, including its actions that cause the deaths of enemy combatants.
There are no other plausible candidates for whom we might hold morally responsible for the deaths of enemy combatants caused by AWS.
Therefore, there is no one whom we may justly hold responsible for the deaths of enemy combatants caused by AWS.
Therefore, it is impermissible to wage war through the use of AWS. To do so would be to “treat our enemy like vermin, as though they may be exterminated without moral regard at all” (2007: 67).
The more likely it is that AWS will behave badly, the more compelling Sparrow’s argument is. Thus it is worth reviewing the formidable difficulties with designing an artificially intelligent system capable of acting well during wartime. For example, AWS cannot consistently distinguish between legitimate and illegitimate targets in chaotic environments. Sensors and mapping technologies have a long way to go before they can reliably determine whether a target is carrying a gun or a loaf of bread. Moreover, whether a potential target is an armed combatant or non-combatant depends on complex contextual details. Armed non-combatant forces may be located in an area where there is a known combatant presence. Non-combatant artillery or warships may pass through enemy territory during wartime. AWS will need to use context in order to determine whether, in these cases, these non-combatants constitute legitimate targets or not.8 The ability to distinguish between combatants and non-combatants is still not sufficient to ensure that AWS will successfully discriminate between targets that are liable to attack and those which are not, as required by the jus in bello principles of just war theory and the laws of armed conflict. For whether a combatant is liable to attack further depends on context. For instance, under jus in bello it is illegitimate to target armed combatants who have indicated the intent to surrender. But, in many circumstances, AWS cannot reliably tell whether an armed and injured soldier has indicated a desire to surrender. It will be exceedingly difficult to successfully program robots to make such fine-grained and context-sensitive discriminations in battle. But this is precisely what must be done if AWS are to reliably adhere to the principles of just war (Guarini and Bello 2012).9
Concerns about AWS’s reliability in selecting targets are contingent in two senses. First, these worries might be assuaged by restricting the deployment of AWS to particular domains (Schmitt 2013). These domains might include operations against naval assets (Brutzman et al. 2010), tanks and self-propelled artillery,or aircraft in a given geographical area (Guarini and Bello 2012).10 In these domains, AWS may prove superior to human-operated weaponry in distinguishing between legitimate and illegitimate targets.11 Second, concerns about the ability of AWS to discriminate between legitimate and illegitimate targets hinge on facts about the current state of artificial intelligence technology. In the future, advances in this technology will very likely enable AWS to discriminate between legitimate and illegitimate targets more reliably than human-operated weapons systems.
Strictly speaking, Sparrow’s argument is meant to apply to AWS even if they never make the wrong decision in wartime.12 AWS would supposedly remain problematic because there would be no one in in principle who could be held responsible, should something go wrong. However, it must be admitted that Sparrow’s argument is deprived of most (or all) of its force if we imagine AWS to be perfect moral decision makers. Why should it be a problem that there is no one we could hold responsible if AWS were to make a mistake, if we know they will never actually make a mistake? We may be skeptical that we could ever have such confidence in the abilities of AWS. Still, under the stipulation that AWS are perfect moral decision makers, a possibility we explore below, the force of Sparrow’s worries evaporate.13 Thus, his is ultimately a contingent objection to AWS. We view all other forms of responsibility-based objections against AWS to be similarly contingent in nature.
Before turning to our moral arguments against AWS, it is worth highlighting a potential drawback of any moral objection to AWS: that it would similarly rule out non-weaponized autonomous technologies such as driverless cars. Previous objections to AWS seem to all have this drawback. Perhaps this is most clear when considering the objection from responsibility gaps. Driverless cars, like AWS, would likely be required to make life and death decisions in the course of operation. Consider, for example, a scenario like the following, first raised by Patrick Lin: “On a narrow road, your robotic car detects an imminent head-on crash with a non-robotic vehicle—a school bus full of kids, or perhaps a carload of teenagers bent on playing ‘chicken’ with you, knowing that your car is programmed to avoid crashes.”14 Suppose, because of its crash-avoidance programming, your car swerves off the road with the result that you are killed. Is your car responsible for your death? Is the manufacturer of the car or its lead programmer responsible? If we prohibit the deployment of AWS on the grounds that they pose difficulties for attributions of moral responsibility, then this presses us to prohibit the deployment of driverless cars for the same reason. This implication will be undesirable to many who see an intuitive moral difference between weaponized and non-weaponized forms of autonomous technology. It would thus be a virtue of an objection to AWS if it could vindicate this intuitive moral difference.15 In the remainder of this essay, we develop a pair of moral objections to AWS that, unlike previous objections, are not contingent. Further, the objections we give against AWS may not rule out non-weaponized autonomous technologies like previous objections would. According to the first objection, AWS cannot, in principle, replicate human moral judgment. According to the second objection, even if AWS could replicate human moral judgment, they could not act for the right reasons in making decisions about life and death.
3 The Anti-codifiability Argument Against AWS
If some form of moral judgment is required for proper moral decision making, and if AWS cannot replicate that judgment, deploying AWS to make such decisions would be morally problematic.
This picture fits only if the virtuous person’s views about how, in general, one should behave are susceptible of codification, in principles apt for serving as major premises in syllogisms of the sort envisaged. But to an unprejudiced eye it should seem quite implausible that any reasonably adult moral outlook admits of any such codification. As Aristotle consistently says, the best generalizations about how one should behave hold only for the most part. If one attempted to reduce one’s conception of what virtue requires to a set of rules, then, however subtle and thoughtful one was in drawing up the code, cases would inevitably turn up in which a mechanical application of the rules would strike one as wrong… (1979: 336, emphasis added)16
Since the appearance of McDowell’s influential piece, philosophers have continued to reject the codifiability thesis for many reasons.17 Some have rejected the view that there are any general moral principles.18 Even if there are general moral principles, they may be so complex or context-sensitive as to be inarticulable.19 Even if they are articulable, a host of eminent ethicists of all stripes have acknowledged the necessity of moral judgment in competently applying such principles.20 This view finds support among virtue ethicists, whose anti-theory sympathies are well storied.21 Mill22 was also careful to acknowledge the role of moral judgment, as have been his intellectual heirs, consequentialists like Scheffler and Hooker.23 Finally, Kant, Ross, and McNaughton are among the deontologists who acknowledge the essential role of moral judgment.24
Thus, many prominent ethicists, spanning a range of popular positions, find the necessity of moral judgment for proper moral decision making plausible. We wish to remain agnostic on the particular species of judgment that is required to successfully follow the true moral theory. It may be that the exercise of moral judgment has a necessary phenomenal character or ‘what-it’s-like’. It could be that successfully following the true moral theory requires a kind of practical wisdom. It could be that a kind of wide reflective equilibrium is needed, which requires us to strike the right balance between general moral principles and our moral intuitions.25 All that is required for our argument against AWS is that one of these accounts of moral judgment, or something similar, offers the right picture of moral judgment.
Programmed behavior is either arbitrary or strictly rulelike. Therefore, in confronting a new usage a machine must either treat it as a clear case falling under the rules, or take a blind stab. A native speaker feels he has a third alternative. He can recognize the usage as odd, not falling under the rules, and yet he can make sense of it—give it a meaning in the context of human life in an apparently nonrulelike and yet nonarbitrary way. (1992: 199)26
Similarly, because moral deliberation is neither strictly rulelike nor arbitrary, ‘programmed behavior’ could never adequately replicate it (at least in difficult cases). Furthermore, take the possible requirements of moral judgment considered above: phenomenal quality, phronesis, and wide reflective equilibrium. Only a minority of philosophers of mind believe that AI could have phenomenal consciousness—most are skeptical or uncommitted. If AI cannot be conscious in this way, and if this kind of consciousness is what moral judgment requires, then AI will never be able to engage in moral judgment. It is also plausible that an artificial intelligence will never be able to exercise practical wisdom of the kind possessed by the phronimos. And since artificial intelligences cannot have intuitions, they cannot engage in wide reflective equilibrium. Since it seems likely that an artificial intelligence could never possess phenomenal consciousness, phronesis, or the intuitions required for wide reflective equilibrium, it seems unlikely that AI will be able to engage in any kind of moral judgment.
Hence, we could never trust an artificial intelligence to make a moral decisions, and so we should expect them to make significant moral mistakes. For example, it would be far easier to make AI carry out immoral or criminal orders than it is to get human soldiers to carry out such orders. If an AWS cannot make moral judgments, they cannot resist an immoral order in the way that a human soldier might, because they are incapable of evaluating the deontic status of the order.27 It is not just that an AWS would be more prone to making moral mistakes. Rather, we argue, they could not in principle discern the correct answer. Unless the true moral theory is codifiable, artificial intelligence can never be trusted to make sound moral decisions.
4 Objections and Responses
An opponent might object, “Earlier you purported that your first argument would be superior to the contingent arguments that have come before. But isn’t it contingent whether following the true moral theory requires the exercise of judgment, and whether an artificial intelligence could replicate that moral judgment? Isn’t the substrate independence of minds, for example, contingently false, if it is false?”
By way of response, note that our argument is based on two claims: (1) that the structure of the true moral theory is not codifiable, and thus that a particular set of psychological capacities is required to successfully follow the true moral theory; and (2) that an artificial intelligence could not in principle manifest these capacities. It is widely believed that each of these claims is, if true, necessarily true. It is relatively uncontroversial among ethicists that the true moral theory has its structure necessarily. Our second claim is more contentious, but also widely held among philosophers of mind. If any of such views is true, including reductive type physicalism or other views that hold that consciousness depends on a biological substrate, then (2) is true.28 Moreover, the metaphysical requirements of phenomenal conscious are necessary, such that if any of these theories is true, it is necessarily true. While it is possible that what we say in this section is mistaken, unlike other objections, our worries about AWS are not grounded in merely contingent facts about the current state of AI. Rather, if we are correct, our claims hold for future iterations of AI into perpetuity, and hold in every possible world where there are minds and morality.
We will now take care in spelling out two rather serious objections to our argument up to this point. We believe there are satisfactory responses to both and, since the objections are related, we respond to both at the same time below.
First, consider the following objection. “Your argument does not show that it would be morally bad to employ AWS, even granting that they would be worse than humans at making moral decisions. This is because they would be better than humans at carrying out the decisions they do make correctly, for example, they would be better at targeting. In the end, it might be worth the moral cost to deploy AWS if they make comparatively few mistakes in decision making while being better at executing a decision when it is the right one.”
We will spell out this objection in some detail, since its rejection leads naturally to our second argument against AWS. Consider three kinds of mistakes that we might make in performing actions with moral import. First, there are what we will call empirical mistakes. These are mistakes we might make in discovering and identifying the empirical facts that are relevant to our moral decision making. For example, it would be an empirical mistake to believe a target is carrying a gun when the object is in fact a camera. Second, there are genuine moral mistakes, which are mistakes in moral judgment, e.g., about what the relevant moral considerations are or how to weigh the relevant moral considerations once they have been discovered. These mistakes occur when we come to the wrong normative answer about a moral problem, even given full information about the descriptive facts. Finally, there are what we will call practical mistakes, which occur when we have made the right decision, informed by the right considerations, but have nonetheless made a mistake in acting on our moral judgment, for example, by reacting a moment too slowly, or by missing one’s target and shooting an innocent person due to mental fatigue.
There is good reason for thinking that AWS could commit drastically fewer empirical and practical mistakes than human soldiers. Decisions on the battlefield must incorporate massive amounts of data, and they must be made in seconds. Adams (2001) points out that the tempo of warfare has increased dramatically in recent years, and so it will presumably only accelerate further. The human mind is only capable of incorporating so much information and acting on it so quickly. The day may come when human combatants simply cannot respond quickly enough to operate effectively in modern warfare. AWS has the potential to incorporate massive amounts of information, thereby avoiding the empirical and practical mistakes that humans are eventually bound to make in the ever-quickening pace of battle.29
Now suppose that we are right in our arguments above, that AWS will never be able to replicate human judgment because the true moral theory is not codifiable. If this is true, then one could predict that AWS might be prone to make more genuine moral mistakes than humans. More often than human combatants, they would fail to derive the morally correct conclusion from the descriptive facts. But, at the same time, AWS might be less likely to make empirical and practical mistakes.30 This might have the result that AWS are better than manned weapons systems at achieving morally desirable results in the long run. (Suppose, for example, that AWS successfully avoid killing several non-combatants that a human would have killed due to empirical or practical mistakes.) Obviously, more about the practical abilities and machine learning capabilities of AWS would need to be said to justify this last claim, but let’s assume it is true for the sake of argument. This would mean that, even if the anti-codifiability thesis is true, we would have to put disproportionate moral weight on genuine mistakes in moral judgment in order to generate the verdict that it was problematic to deploy AWS in the place of humans.
A second and related objection asks, simply enough, what if the anti-codifiability thesis is false? What if computers could become as good as or better than humans at making moral decisions or, indeed, could become perfect at making moral decisions? Suppose AI could pass a kind of Turing test for moral reasoning.31 If so, the entire argument against AWS would seem to be invalidated.
Each of these objections stems from the intuitive thought that as long as AWS will someday manifest behavior on the battlefield that is morally superior to human behavior, there is surely no objection to be found to their deployment.
5 Acting for the Right Reasons: A Second Argument Against AWS
If the anti-codifiability thesis is false then our first objection to AWS fails. Even if the anti-codifiability thesis is true, our first objection to AWS succeeds only if we place disproportionate disvalue on genuine moral mistakes compared with empirical and practical mistakes. Our second objection to the deployment of AWS supposes that AI could become as good as or better than humans at making moral decisions, but contends that their decisions would be morally deficient in the following respect: they could not be made for the right reasons. This provides the missing theoretical basis for the disproportionate disvalue that our first argument places on genuine moral mistakes.32
Imagine a racist man who viscerally hates all people of a certain ethnicity and longs to murder them, but he knows he would not be able to get away with this under normal conditions. It then comes about that the nation-state of which this man is a citizen has a just cause for war: they are defending themselves from invasion by an aggressive, neighboring state. It so happens that this invading state’s population is primarily composed of the ethnicity that the racist man hates. The racist man joins the army and eagerly goes to war, where he proceeds to kill scores of enemy soldiers of the ethnicity he so hates. Assume that he abides by the jus in bello rules of combatant distinction and proportionality, yet not for moral reasons. Rather, the reason for every enemy soldier he kills is his vile, racist intent.
We contend that it would be wrong to deploy the Racist Soldier, other things being equal, knowing his racist tendencies and desires. That is, if we had a choice between deploying either Racist Soldier or another soldier who would not kill for such reasons, and both would accomplish the military objective, we would have a strong moral reason to choose the non-racist soldier. The likely explanation for this is that, while Racist Soldier abides by the constraints of jus in bello, he is acting for the wrong reasons. We believe this judgment can be extended to AWS. Just as it would be wrong to deploy the Racist Soldier, it would be wrong to deploy AWS to the theater of war because AWS would not be acting for the right reasons in making decisions about life and death.33
If either the desire-belief model or the predominant taking as a reason model of acting for a reason is true, then AI cannot in principle act for reasons.34 Each of these models ultimately requires that an agent possess an attitude of belief or desire (or some further propositional attitude) in order to act for a reason.35 AI possesses neither of these features of ordinary human agents. AI mimics human moral behavior, but cannot take a moral consideration such as a child’s suffering to be a reason for acting. AI cannot be motivated to act morally; it simply manifests an automated response which is entirely determined by the list of rules that it is programmed to follow. Therefore, AI cannot act for reasons, in this sense. Because AI cannot act for reasons, it cannot act for the right reasons.36
One may here object that Racist Soldier shows only that it is wrong to act for the wrong reasons. It does not establish the positive claim, asserted above, that there is something morally problematic about failing to act for the right reasons. As we have just suggested, it is not the case that the AI is acting for the wrong reasons (as the racist soldier is), but rather the AI is not acting for any reasons at all. This means that if our argument against the deployment of AI is to work, we must establish the positive claim that failing to act for the right reasons is morally problematic as well.
Imagine a sociopath who is completely unmoved by the harm he causes to other people. He is not a sadist; he does not derive pleasure from harming others. He simply does not take the fact that an act would harm someone as a reason against performing the act. In other words, he is incapable of acting for moral reasons.37 It then comes about that the nation-state of which this man is a citizen has a just cause for war: they are defending themselves from invasion by an aggressive, neighboring state. It so happens that the man joins the army (perhaps due to a love of following orders) and eagerly goes to war, where he proceeds to kill scores of enemy soldiers without any recognition that their suffering is morally bad. He is effective precisely because he is unmoved by the harm that he causes and because he is good at following direct orders. Assume that he abides by the classic jus in bello rules of combatant distinction and proportionality, yet not for moral reasons. No, the sociopathic soldier is able to operate effectively in combat precisely because of his inability to act for moral reasons.38
Most who think it would be morally problematic to deploy the racist soldier in virtue of the fact that he would be acting for the wrong reasons will also think it would be clearly morally problematic to deploy the sociopathic soldier over a non-sociopathic soldier. If there is a moral problem with deploying the sociopathic soldier, however, it is most plausibly derived from the fact that he would fail to act for the right reasons.39 But we have already established that AWS cannot, in principle, act for reasons in the relevant sense, and thus that they cannot act for the right reasons. The actions performed by AWS in war will therefore be morally problematic in the same way as the sociopath soldier: neither of them acts for the right reasons in killing enemy combatants.40
A state must intend to fight the war only for the sake of its just cause. Having the right reason for launching a war is not enough: the actual motivation behind the resort to war must also be morally appropriate. Ulterior motives, such as a power or land grab, or irrational motives, such as revenge or ethnic hatred, are ruled out. The only right intention allowed is to see the just cause for resorting to war secured and consolidated. If another intention crowds in, moral corruption sets in. (Orend 2008).
It is possible, and meaningful, to criticize some of the non-moral motives which states can have in going to war while still endorsing the moral motive. But that motive must be present: Walzer concurs that right intention towards just cause is a necessary aspect of the justice of resorting to war. This is to say that it must be part of a state’s subjective intention in pursuing war that it secure and fulfil the objective reason which gives it justification for fighting. (Orend 2006: 46).
According to Orend, a state is in greatest violation of the criterion of right intention when its motivational set fails to contain the relevant moral considerations which justify the resort to war. Thus the jus ad bellum criterion of right intention already imposes a positive moral demand on actors to act for the right reasons in reaching the decision to go to war.
What we have said so far about the moral relevance of intentions has been an appeal to various authorities. These theorists might be mistaken about their interpretation of the criterion of right intention. Are there independent reasons for thinking that morality requires agents (including states and individual soldiers) to possess certain motivations in order for their actions to count as morally right or just? One reason to prefer the positive interpretation of the criterion of right intention is that the negative interpretation would make it needlessly difficult for states to justify wars that we think should be justified, such as third-party defense against aggression or humanitarian intervention. Suppose that country X decides to resort to war against country Y because Y has initiated military aggression against country Z. X’s reason for going to war against Y is that doing so would constitute the resistance of aggression. Now suppose that after X has decided to resort to war against Y on the basis of this reason, but before it has initiated military deployment, it is discovered that Z possesses valuable resources which are highly coveted by X. X realizes that defending Z against aggression from Y would put X in a desirable bargaining position with respect to trade with Z. Realizing this, X adopts a further reason for resorting to war against Y; that resorting to war would improve X’s future bargaining position with respect to trade with Z. This further reason, it is safe to say, is a ‘bad’ reason to resort to war in the following sense. If this were X’s only reason for resorting to war, the resort would, plausibly, be unjust. But it is stipulated that X has been moved to resort to war for reasons that appear, on their own, sufficient to justify the resort to war against Y (that doing so would constitute the resistance of unjust aggression). Clearly, X’s third-party defense against aggression cannot be rendered unjust by the mere fact that it discovered new information and subsequently added a new reason to its motivational set. It may here be objected that X’s further selfish reason for resorting to war does not render its resort to war unjust because that reason is inefficacious. X has already decided to resort to war for the right reasons, so adding this wrong reason makes no difference to X’s actions. Notice that this objection concedes that the presence of the right reasons makes a morally significant difference to the moral status of a resort to war, and that is all we are after. We should understand the criterion of right intention as a requirement to act for the right reasons rather than as a mere prohibition against acting for the wrong ones.
love of violence, revengeful cruelty, fierce and implacable enmity, wild resistance, and the lust of power, and such like… it is generally to punish these things, when force is required to inflict the punishment, that, in obedience to God or some lawful authority, good men undertake wars… (1887)
Augustine’s language might lead us to think he is referring to the jus ad bellum criterion of right intention by discussing the causes for which men undertake, i.e., begin, wars. However, this interpretation would be faulty, since Augustine speaks of men who undertake wars “in obedience to some lawful authority,” and this can only mean soldiers themselves. Otherwise, Augustine should refer to the reasons for which the lawful authorities undertake wars, but here he does not. Thus, Augustine believes there are moral requirements for soldiers themselves to act for the right reasons. Aquinas, for his part, quotes Augustine approvingly in his Summa Theologica (1920). According to Reichberg, Aquinas is principally concerned with “the inner dispositions that should guide our conduct in war” (2010: 264).
Prominent contemporary just war theorists have agreed. Thomas Nagel (1972: 139), Robert Sparrow (2007: 67–68), and Peter Asaro (2012) have acknowledged the plausibility of this view—with roots in the historical just war tradition—that intentions matter morally not just for policymakers ad bellum but for soldiers in bello as well.43 Sparrow (unpublished manuscript) is the most compelling defense of this view with which are familiar.44
Given the historical and contemporary concern with combatant intentions in Aquinas, Augustine, Nagel, McMahan, Sparrow, and Asaro, it is worth asking why this concern is not encoded in contemporary just war theory as a jus in bello criterion of right intention. We suspect that the absence of such a criterion is explained by the epistemic difficulty in discerning combatant intentions. Considering the difficulties associated with identifying a state’s intentions in resorting to war, it may seem hopeless to attempt to determine the reasons that move an individual soldier to carry out wartime maneuvers.45 However, if epistemic difficulty best explains the omission of a jus in bello criterion of right intention, this difficulty does not apply to the case of AWS. Whereas the reasons for which human combatants kill in war are opaque—only the soldiers themselves have access to their reasons—we have argued that AWS cannot in principle act for reasons, and so it is guaranteed that they will fail to act for the right reasons in deciding matters of life and death. Hence, there is no similar epistemic problem for AWS. Just as states should not resort to war without doing so for morally admirable reasons, wars should not be fought by soldiers who cannot in principle kill for admirable reasons.
Consider, first, giving flowers to Mary only in order to cheer her up, as opposed to doing so merely to make Mary’s boyfriend jealous. Although the two actions are alike in one respect—both involve giving Mary a gift— the different ends make for a difference in the actions’ nature and value. Only the former is acting generously, while the latter is acting spitefully. In one sense the intended end is “extrinsic” to the action: one can have and intend an end independently of, and prior to, performing the action, and the action can be described without any reference to the intended end. And yet something extrinsic to an act can nevertheless transform the act from merely giving flowers into the realization of acting generously (or spitefully), which has a distinctive value (or disvalue) (Jaworska and Tannenbaum 2014: 245).46
It appears in Jaworska’s and Tannebaum’s example that the agent’s motivation to cheer up Mary actually confers moral value on the action that would otherwise be absent. This conferral of value will, on at least some occasions, be sufficient to make the moral difference between an action that is morally permitted and one that is morally prohibited. For instance, it is plausible that it would be permissible to give Mary flowers with the intended ends both of making her boyfriend jealous and cheering her up, while it would be impermissible to give her flowers merely in order to make her boyfriend jealous.
Further support for the positive interpretation of the criterion of right intention can be found in Thomas Hurka’s discussion of the deontic significance of agents’ motivating attitudes (Hurka 2010: 67).47 Hurka’s account is complicated, and a complete defense of his position is beyond the scope of this paper. For our purposes it will suffice to focus on his remarks about the moral significance of lacking motivation (i.e., taking an attitude of indifference). Let us suppose that failure to be motivated by some consideration of which one is aware is tantamount to manifesting an attitude of indifference toward that consideration. The attitude of indifference is not, on its own, morally laden. Like the attitudes of taking pleasure or pain in some state of affairs, the moral import of indifference depends on the object toward which it is taken. If one is indifferent toward the fact that one’s glass is half empty, this attitude does not have any obvious moral import. However, if one is indifferent toward the fact that there is a young child wandering into traffic, one’s attitude takes on a very different moral significance. In Hurka’s words, “Complete indifference to another’s intense suffering is callous, and callousness is not just the lack of a virtue; it is a vice and therefore evil” (Hurka 2010: 66). One’s utter indifference to the child’s potential suffering (i.e., one’s utter failure to be motivated by his plight) is downright evil.
Suppose that one fails to remove the child from the road because one is indifferent toward his plight. It is tempting to say that one’s attitude of indifference in part explains why one’s failure to save the child is morally wrong. We believe that Hurka’s understanding of the moral significance of motivations generally, and the attitude of indifference in particular, supports the positive interpretation of the criterion of right intention and its extension to individual soldiers.
We have shown (i) that there is plausibly a moral objection to deploying a human sociopath soldier who would successfully carry out his wartime duties, but not for the right reasons, (ii) that there is a (justified) precedence in just war theory for a positive requirement to act for the right reasons in decisions about going to war, (iii) that just war theorists past and present have attributed deontic significance to the reasons for which individual soldiers act in participating in war, and (iv) that the epistemic problems with discerning human soldier’s reasons for killing enemy combatants do not apply to AWS. These considerations dictate that the objection to AWS that we are presently considering is simply a logical extension of just war theory’s legitimate concern with acting for the right reasons in deciding matters of life and death.
One might at this point concede (i) and (ii), but object that there is an important difference between the soldiers in our earlier examples and AWS. Unlike the soldiers, the failure of AWS to act for the right reasons is due to the fact that AWS are not acting at all (Sparrow unpublished manuscript). AWS are simply sophisticated guided cruise missiles or landmines, and like cruise missiles or landmines, are not really agents at all.48 It would be absurd to insist that a cruise missile or a landmine must possess the right intention in order for it to be permissible to deploy it in war, because a cruise missile is not acting in locking onto its target. One might in this way concede that we are right that AWS are not acting for the right reasons, but contend that this is because talk of them acting at all is nonsense.
This is one of the most enduring problems for objections to AWS, and our argument above is not spared. But remember we are here discussing highly autonomous weapons that are actually making decisions. Surely an AWS is not totally inert; its purpose is precisely to make decisions about who should live or die; to discriminate on its own between targets and courses of action; indeed, to fulfill all of the purposes that a soldier would fulfill in its place. This objection characterizes AWS as if they were mere landmines, cruise missiles or bullets. But if a bullet or a landmine were choosing its targets, it would be a very different bullet indeed.49
Furthermore, we may wonder what reasons we have for doubting that AWS can act in the relevant sense. One line of support for this doubt begins with the observation that AWS are not full blown agents. This might actually be entailed by our claim that AWS cannot respond to reasons. It is tempting to infer from the fact that AWS are not responsive to reasons—and thus not agents—that AWS cannot act at all. This inference is too quick, however. On many plausible accounts of reasons-responsiveness—and given certain assumptions about the capacities of most non-human animals—most non-human animals are not responsive to reasons either. But no one doubts that non-human animals can act. The conviction that AWS cannot act is rendered suspect insofar as its primary support comes from the thought that acting requires agency in the form of responsiveness to reasons.
Finally, if it turns out that autonomous weapons are no different with respect to the capacity for action or agency than bullets, cruise missiles or landmines, then we are open to simply conceding that there may not be any non-contingent moral problems with autonomous weapons. For it is this apparently distinctive feature of autonomous weapons—that they make decisions about which individuals to target for annihilation which they then act on—which seems to ground the common moral aversion to their deployment. If it turns out that we are mistaken about AWS possessing the capacity to choose their targets then perhaps this is a reason to accept that our common moral aversion to their deployment is mistaken as well.
We close this section by acknowledging important limitations of the above argument—and perhaps any argument—against AWS. We have only attempted to show that there is something seriously morally deficient about the way that AWS go about making decisions about ending human lives. In other words, we have defended the existence of a powerful pro tanto moral reason not to deploy AWS in war. We have not shown that this reason is decisive in the face of all countervailing moral considerations. For example, if deploying AWS in a particular conflict can be expected to reduce civilian casualties from 10,000 to 1000, this consideration might very well override the fact that AWS would not act for the right reasons in achieving this morally desirable result. Indeed, if AWS prove to be sufficiently superior to traditional armed forces at achieving morally desirable aims in war, then there may not be any moral objection strong enough to render their deployment morally impermissible.50 Still, until we are confident in such a marked superiority, we consider this pro tanto reason to pose a significant obstacle to their deployment.
6 Non-weaponized Autonomous Technology
Any account of the permissibility of autonomous weapons systems will risk prohibiting the use of autonomous decision making technologies that most people view as neutral or morally good. While many of us tend to have a significant moral aversion to the thought of autonomous weapon systems, most have no such similar moral aversion to non-weaponized autonomous systems such as driverless cars. In fact, for many people, the opposite is true: many of us hold that non-weaponized future autonomous technology holds the potential for great good in the world.51 While the prospect of driverless cars raises interesting ethical challenges of its own,52 virtually no one is inclined to posit that driverless cars are on the same shaky moral ground as autonomous weapons. Yet this—lumping driverless cars and AWS together—seems to follow from all the contingent and responsibility-based objections to AWS currently on offer. Thus we have to be careful not to throw out the baby with the bathwater.
What is the basis for this different moral reaction to an otherwise similar technology? One answer is that there is no basis for this reaction, and autonomous cars are no more or less morally problematic than AWS. It is certainly possible that people simply lack the education and technical understanding to grasp the moral issues that are at stake with driverless cars, just like a majority of people fails to appreciate understand the wider implications of automated surveillance affecting individual freedom and autonomy.53 That is, it is entirely possible that the arguments given in this paper could very well apply to driverless cars; it would not constitute a reductio ad absurdum against those arguments in that case. We must weigh the costs. If the moral worries we’ve raised here are a problem for driverless cars, it could simply be a moral cost that is outweighed by the significant moral gains such technology would portend. It is unclear. In any case, while we think it is a legitimate possibility—one that we do not rule out—that moral objections to AWS equally impugn non-weaponized autonomous technology, it would be preferable to avoid it.54 Hence we will attempt to identify a legitimate moral difference between weaponized and non-weaponized autonomous technologies.
It is tempting to answer that an autonomous weapon would be making certain moral judgments that we think are only fitting for a human moral agent to make, while the non-weaponized autonomous robots are not involved in such judgments. But there is a tremendous—and underappreciated—difficulty with this response.
For, notice, scenarios could arise in which a non-weaponized autonomous robot would need to make similar, morally weighty, life-and-death choices that require a moral judgment. Imagine: A small child unexpectedly appears on the road ahead of a driverless car. The car quickly determines that if it tries to swerve it will likely cause a crash and thus kill the occupant of the car. Yet, if the car does not serve, it will almost surely kill the child.55 What should the car do?56 Whatever one thinks it should do, this involves a morally weighty judgment, similar to the kinds of moral judgments that would be required by (and what we presumably find morally problematic about) the kind of autonomous weapon systems we are imagining.
Our response is twofold. First, AWS are designed with the purpose of making moral decisions about human life and death, whereas driverless cars are intended for a wholly different (and peaceful) purpose. If they should end up making moral decisions about life and death, it is merely foreseen but not intended. Return to the racist soldier case. Only suppose that, instead of deploying Racist Soldier to the front lines of combat where we know he will encounter (and kill) members of the group he hates, we deploy him to a different front where his exposure to members of the group he hates is very unlikely. This seems morally acceptable even if there is a small chance that he will still encounter and kill a member of that group in this different front, despite our efforts to avoid this result. Were this to happen, it would be regrettable, but this circumstance would have been merely foreseen but not intended. Morally, we have made a very different—and less worrisome—choice than if we had purposefully put him somewhere with the intention that he kill people.
Though the moral significance of the distinction between intending and foreseeing is firmly entrenched in just war theory, its moral significance is controversial among contemporary ethicists. Our second reply rests on less controversial ground. Notice that AWS will, as a matter of fact, constantly make life-and-death decisions regarding humans, if they are doing their job. That is, the probability of AWS making life-and-death moral decisions is very high given that the capacity to make such decisions is the explicit reason for which they are deployed and put into use. These decisions can be expected to be radically less common with driverless cars, since that is not the reason they would be put into use.57
Notice that we could again call on the Racist Soldier case, just as we did for the ‘foreseeable but unintended’ response. Suppose we deploy Racist Soldier to a theater of war where he will have a greatly reduced likelihood of encountering and killing a member of the group he hates. This seems morally acceptable even though there remains a small chance that he will encounter and kill a member of that group. But we accept this because it is much less likely than if he were deployed to a more active theater of war.
The distinction between intending and foreseeing might also go some way toward justifying the limited deployment of AWS in defensive contexts where they are unlikely to do harm to human life. Autonomous missile defense systems might be one example. Of course there would remain some risk of harm to human life, but the risk would be small and merely foreseen so long as they are deployed in limited contexts. Deploying AWS in limited defensive contexts might avoid the moral problems with deploying AWS in offensive contexts.58
Unlike AWS, driverless cars, and some forms of autonomous missile defense systems, are not deployed with the intention that they will make life and death decisions. Nor are they nearly as likely as AWS to need to make life and death decisions. We find these two responses to be at least partially satisfying in conjunction. Even if the responses fail to maintain a hard moral distinction between weaponized and non-weaponized AWS, however, we are not ultimately concerned about our argument ruling out driverless cars and other autonomous systems. We ought to meet a high bar before deploying artificial intelligences of any kind that could make morally serious decisions—especially those concerning life and death. It is plausible that no autonomous system could meet this bar.
Imagining a future of autonomous weapons like those we describe above poses other challenges. Suppose autonomous weapons become genuine moral decision makers, i.e., they become agents. If all agents—even artificially intelligent ones—have equal moral worth, then a strong motivation for deploying AWS in place of human soldiers, i.e., the preservation of morally important life, becomes moot.59
Suppose, as many virtue theorists do, that acting for the right reasons is a necessary constituent of a good life. If, as we maintain, AWS and other AI cannot act for the right reasons, then bringing them into existence would mean bringing into existence an agent while denying it the possibility of a good life.
There is also a Meno problem for moral machines.60 A sufficiently advanced artificial intelligence could seem radically alien, and have an equally exotic conscience. Intelligent machines whose moral decisions differ from ours could seem to some to be moral monsters and to others to be moral saints. Judging which of these appraisals is correct will be challenging, but it could well determine the future of the human race.
Suppose this problem is less threatening than it seems and that autonomous weapons eventually become much better than humans at making moral decisions. Wouldn’t it then—obviously—become obligatory to actually surrender our moral decision making to AI?61 This would include not merely decisions made in war, but decisions over whether to go to war. Why should we have human parliaments and congresses, so notoriously bad at making moral decisions, when we could have the AI make such decisions for us? And, at that point, it’s worth asking: why stop at employing AWS in times of war? Indeed, some will think that decisions made about healthcare policies or economic distribution and the like are morally more important than even decisions about war, as it is possible that significantly more people are affected by such actions. Why stop there? Rather, we could be obligated to ‘outsource’ all of our morally important decisions to AI, even personal ones such as decisions about where to live, what career to pursue, whom to marry, and so forth. After all, all of these decisions can easily have consequences that are morally significant. Some people, of course, will be perfectly happy with such a vision. Others confess a deep-seated discomfort with the idea; a discomfort the source of which we have been at pains to investigate. Whatever that reason is that counts against us surrendering all of our moral autonomy to AI, it also counts against us deploying AWS.
It could be that we are uncomfortable with AWS making decisions so easily, in the same way we are uncomfortable with deploying the psychopathic soldier, even supposing he performs all the right actions. We regard with great pity those who have their autonomy co-opted by or outsourced to someone else,62 since we view autonomy as a supremely important good for humans.63 There is something truly disturbing about someone whose life is entirely determined by decisions that are outside of his immediate control. Could we be obligated to enter this pitiable state? If we are resistant, it could be that we ultimately believe that grappling with difficult moral issues is one of the things that gives human life meaning.
The so called Responsibility Gap objection AWS has been developed by several scholars. The exact provenance of responsibility based objections against AWS is debated. In our view, most famously and influentially, Robert Sparrow (2007) argued that if an AWS made a mistake in war and, say, killed a noncombatant, that no one could legitimately be held morally responsible (not the commander, not the programmers, and so forth), resulting in an odd responsibility gap, the possibility of which makes deployment of AWS morally impermissible. This is discussed below. Several others have made a similar point or developed responsibility based objections to AWS. Andreas Matthias (2004) actually coined the term ‘Responsibility Gap’ as it pertains to AWS. Heather M. Roff (2013) has made similar arguments, particularly as it relates to the technical aspects of control. Christian Enemark (2013) has discussed similar arguments. Alex Leveringhaus has argued that responsibility based objections fail to rule out AWS as morally impermissible (2013).
The autonomous weapons we have in mind are an example of “weak AI”: they boast sophisticated decision- making abilities, even to the extent that their ultimate decisions could be a mystery to their creators. But these capabilities are confined to a narrow domain of decision-making, unlike the capabilities of strong AI. The autonomous weapons we have in mind are not good at chess, they cannot play “Jeopardy!”, they cannot diagnose a medical condition from a list of symptoms, and they cannot pass the Turing test in conversation with a human.
To be clear, in our view the kinds of weapon technology in use today does not yet constitute what we mean by AWS, but several weapons point to the impending likelihood of AWS being developed and deployed. The most notable example in widespread use is likely the Phalanx CIWS (close in weapon system) used on US Navy and Royal Navy surface vessels, and it’s land-based variant the C-RAM (Counter Rocket Artillary and Mortar), when those systems are used in so-called ‘autonomous’ mode. But in this paper we are analyzing weapon systems that go beyond such present-day technology.
Of course, our arguments here would apply to many autonomous weapons at lower levels of autonomy as well.
There are several arguments that suggest that fully autonomous weapons will be deployed in the future (Sparrow 2007: 64). See, relatedly, the ‘Principle of Unnecessary Risk’ discussed by one of us, Strawser (2010: 344): “If X gives Y an order to accomplish good goal G, then X has an obligation, other things being equal, to choose a means to accomplish G that does not violate the demands of justice, make the world worse, or expose Y to potentially lethal risk unless incurring such risk aids in the accomplishment of G in some way that cannot be gained via less risky means.” While Strawser (2010) uses this premise in an argument for the obligation to deploy unmanned aerial vehicles, there is clearly an analogous argument to be made for the moral obligation to deploy fully autonomous weapons. We find these arguments compelling, but a fuller exploration is beyond the scope of this paper.
Of course, there are also reasons for militaries to be apprehensive about the deployment of autonomous weapons, namely, precisely that they are autonomous and therefore more difficult to control than human soldiers. We thank an anonymous referee for raising this point. Nevertheless, we believe that armies will face an increasing pressure to outsource the decisions of human soldiers to AWS. In particular, the corresponding decreased risk to our soldiers’ lives (and thus the decreased political cost of waging war), combined with the increased accuracy and reliability of AWS in some domains, will make their deployment an irresistible option.
This may not pose a decisive practical problem for AWS. In reality, many accepted practices of warfare such as bombing do not provide the option of surrender and do not require stopping an attack when somebody gets injured. Thank you to an anonymous referee for making this point.
Ronald Arkin has made similar points in conversation and conference presentations. Also see Arkin (2009).
We are heavily indebted to Sparrow (unpublished manuscript), for alerting us to these possible solutions.
As Sparrow puts it, even if AWS never commit an action “of the sort that would normally be described as a war crime” (2007: 66).
Supposing that AWS would become just as reliable as humans at making moral decisions (and not more), we generate another interesting worry that has not appeared in the literature. In these cases, we might encounter the inverse of Sparrow’s responsibility gaps, namely, merit gaps. Whereas Sparrow worries that we would have no one to blame or punish in the event a robot makes a mistake, we might just as well worry that we would have no one to praise or reward should an autonomous weapons system perform especially admirably. Worries about such a merit gap seem much less serious, and we wonder if this points either to an asymmetry in our ascriptions of praise and blame in general or else an inconsistency in our attribution of agency to autonomous systems. At any rate, such a discussion is outside the scope of this paper.
It would be a virtue but is not required, of course. That is, it may well be unavoidable that any legitimate moral objection against AWS also indicts non-weaponized autonomous technology, though we are hopeful that our objections offered here do not do that for the reasons given below.
See Louden (1992) for an influential account of “moral theorists” as writers who see the project of moral philosophy as including the development of a straightforwardly applicable moral code. What we are calling the necessity of moral judgment is the denial of Louden’s fourth tenet: “The correct method for reaching the one right answer [in some morally freighted situation] involves a computational decision procedure…” (1992: 8).
See McKeever and Ridge (2005) for an excellent cataloguing of the various species of anti-theory.
Dancy (1993) is the most famous proponent of this view.
This view represents the legacy of McDowell’s passage quoted above. See Little (2000: 280): “there is no cashing out in finite or helpful prepositional form the context on which the moral meaning depends.” See also McNaughton, who says that moral principles are “at best useless, and at worst a hindrance” (1988: 191).
See Little (1997: 75): “The virtuous and nonvirtuous can alike believe that cruelty is bad, or conclude that some particular action is now called for. The virtuous person, however, holds the belief as part and parcel of the broad, uncodifiable, practical conception of how to live, while the nonvirtuous person holds it without so subsuming it. The two differ, if you like, in their conceptual gestalts of the situation… Virtue theory, then, does indeed claim that the virtuous person is in a cognitive state—a state satisfying a belief direction of fit—that guarantees moral motivation. But the guarantee is not located in any particular belief or piece of propositional knowledge. It is, instead, located in a way of conceiving a situation under the auspices of a broad conception of how to live.” See also Hursthouse (1995). This intellectual lineage also includes McDowell, whose aforementioned piece defends the Socratic thesis that virtue is a kind of knowledge.
“It is not the fault of any creed, but of the complicated nature of human affairs, that rules of conduct cannot be so framed as to require no exceptions, and that hardly any kind of action can safely be laid down as either always obligatory or always condemnable. There is no ethical creed which does not temper the rigidity of its laws, by giving a certain latitude, under the moral responsibility of the agent, for accommodation to peculiarities of circumstances…” (Mill 1863: 36).
See Scheffler (1992: 43): “If acceptance of the idea of a moral theory committed one to the in-principal availability of a moral decision procedure, then what would commit one to is something along these lines. But even if it did so commit one, and even if it also committed one to thinking that it would be desirable for people to use such a procedure, it still would not commit one to thinking it either possible or desirable to eliminate the roles played in moral reasoning and decision by the faculties of moral sensitivity, perception, imagination, and judgment. On the contrary, a decision procedure of the kind we have described could not be put into operation without those faculties.” See Hooker (2000: 88): “Rule-consequentialists are as aware as anyone that figuring out whether a rule applies can require not merely attention to detail, but also sensitivity, imagination, interpretation, and judgment”. See also his (2000: 128–129, 133–134, 136).
See Vodehnal (2010: 28 n53): “On a Kantian account, significant amounts of moral judgment are required to formulate the maxim on which an agent intends to act, and which the agent can test using the Categorical Imperative. In addition, the entire class of imperfect duties leaves agents with extensive latitude in how these duties are fulfilled, requiring significant moral judgment as well.” See Ross (2002: 19): “When I am in a situation, as perhaps I always am, in which more than one of these prima facie duties is incumbent on me, what I have to do is to study the situation as fully as I can until I form the considered opinion (it is never more) that in the circumstances one of them is more incumbent than any other…” (emphasis added). See McNaughton, op. cit.
As James Griffin writes: “The best procedure for ethics… is the going back and forth between intuitions about fairly specific situations on the one side and the fairly general principles that we formulate to make sense of our moral practice on the other, adjusting either, until eventually we bring them all into coherence. This is, I think, the dominant view about method in ethics nowadays” (Griffin 1993). See also Van den Hoven (1997).
See, specifically, what Dreyfus terms the “epistemic assumption” of the project of artificial intelligence (1992: 189–206). That assumption is that a system of formalized rules could be used by a computer to reproduce a complex human behavior (for our purposes, moral deliberation). This is one of several assumptions underlying the project of artificial intelligence about which Dreyfus is deeply skeptical. On a fascinating tangential note, see the “ontological assumption,” the genus of which the anti-codifiability thesis is a species, and which Dreyfus terms “the deepest assumption underlying… the whole philosophical tradition” (1992: 205).
We thank an anonymous referee for highlighting this example of one kind of moral mistake that AWS might be prone to make.
Some transhumanists argue that we could one day replicate the human brain, and hence the human mind (including intentions, consciousness, and qualia). Advances in quantum computing or nanotechnology could allegedly make this possible. However, the transhumanists like Bostrom (2003) and Kurzweil (2000; 2005) who are confident about this prospect are a small minority, and there are several famous counterexamples to their views that many philosophers of mind take to be conclusive. See: Block (1978), Searle (1992), Schlagel (1999), and Bringsjord (2007), among others. We thank an anonymous referee for this journal for pressing us on this point.
We are here heavily indebted to Sparrow (unpublished manuscript) and much of this last point is rightfully attributed to him. Also consider, for example, the analogous case of driverless cars, which are safer and more efficient ‘drivers’ due in part to their ability to process much more information than a human driver and to react almost instantaneously (Del-Colle 2013).
There are many plausible reasons for this. For example, unlike humans, robots may never become fatigued, bored, distracted, hungry, tired, or irate. Thus they would probably be more efficient actors in a number of contexts common during warfighting. For more discussion on this point, see Arkin (2009).
George R. Lucas (2013) raises similar thoughts along these lines.
This constitutes yet another respect in which our argument is not ultimately contingent.
Our suggestion in this section aligns neatly with—and can be recast in terms of—Julia Markovitz’ account of morally worthy action (Markovitz 2010). Markovitz provides an account of morally worthy action according to which morally worthy actions are those performed for the reasons why they are right. To put our objection to AWS in terms consistent with Markovitz’ account, AWS are morally problematic because they are incapable of performing morally worthy actions.
Scanlon (1998: 58–64), for instance, endorses the view that reason-taking consists basically in the possession of belief-like attitudes about what counts as a reason for acting. Schroeder has proposed that the considerations that one takes as reasons are considerations about the means to one’s ends that strike one with a ‘certain kind of salience’, in the sense that ‘you find yourself thinking about them’ when you think about the action (Schroeder 2007: 156). Schroder’s account seems to require some kind of attitude of holding a consideration before one’s mind. AI cannot manifest either of these attitudes.
Our point in this argument is directly contra to Sparrow (2007: 65), which takes for granted that artificially intelligent systems will have ‘desires,’ ‘beliefs,’ and ‘values,’ at least in some inverted commas sense.
While it is controversial, it is widely held that psychopaths are unable of appreciating characteristically moral reasons in their deliberations. However, the data also support the view that psychopaths are can recognize moral facts, but are simply not moved by them to act morally. See Borg and Sinnott-Armstrong (2013) for a survey of the relevant literature. Here, we suppose the first of these views, and we think this is acceptable since it is supported by some scientific findings.
The unease with which we still regard the sociopathic soldier recalls Williams’ objection to utilitarianism on the grounds that it discounts a person’s integrity. Williams regards the utilitarian’s answer in Jim’s case as “probably right” (Williams 1995: 117). One way of understanding the objection he famously elaborates here is not that utilitarianism gets the answer wrong, it is that it treats the answer as obvious (Williams 1995: 99). We think the sociopathic soldier example lends some credence to Williams’ original argument.
It is worth acknowledging the reality that sociopaths find their way into the military—it is often difficult to screen them out—but we take this fact to be regrettable. Suppose, however, that most will accept this regrettable fact as an unavoidable cost of doing business. There is nonetheless an important difference between foreseeing that some small number of human sociopaths will inevitably find their way into the military and adopting a national policy of deploying large numbers of sociopaths to fight our wars for us. Adopting such a policy is not an inevitable cost of doing business, nor is the deployment of AWS. We thank an anonymous referee for helpful discussion of this point.
Of course there could potentially be some moral benefit to deploying the sociopath or AWS. For instance, neither a sociopath nor a machine is capable of feeling the force of moral dilemmas; they will therefore not suffer psychological harm associated with making difficult moral choices. But this fact, on its own, does not mean that it is preferable to deploy sociopaths or AWS. Mike Robillard and one of us (Strawser) have recently written about a peculiar kind of moral exploitation that some soldiers experience (2014). They argue that when society imposes the difficult moral choices required by war on soldiers who are not properly equipped to handle them, this can result in certain unjust harms to the soldiers; a form of exploitation. While this sounds plausible, one should not thereby conclude that it would be better to use actors incapable of moral feelings in war, such as AWS. Rather, it simply raises the (already significant) burden on society to employ as soldiers only those who are reasonably capable of making the moral decisions necessary in war.
See Augustine’s (2004) Letter 189 to Boniface, §6.
Similarly, one might object that in prosecuting a war individual combatants do not act in a personal capacity but rather, as agents of the state, in a purely “professional” or “official” capacity and, as such, their intentions are not relevant. Such a view is highly controversial among moral philosophers writing about war, and we disagree that such a distinction can rule out the moral relevance of the actors intentions who carry out war (or any ‘official’ policy) for the reasons given below regarding in bello intentions. Consider: we still think the intentions of a police officer are morally relevant in our judgment of an action she may carry out, even if the action is taken as part of her official duties. Our thanks to an anonymous reviewer for help on this point.
What we say here—and quote others as saying—is a defense of the view that having the right intention is necessary for acting rightly. It should go without saying that having the right intention does not guarantee that an agent acts rightly.
We might also consider our reactions to another modification of the racist soldier case, call it the Racist Policy case. Imagine that it were official policy to deploy racist soldiers; take the policies of the Confederacy during the United States Civil War as a hypothetical example. Then, this distinction between personal behavior and official policy becomes blurred, once the racist motives of the soldier are endorsed and underwritten by the state. Considering that, for reasons we mention above, AWS could become widespread, i.e., their deployment could become official policy, even proponents of this more restrictive view have reason to be alarmed. We are grateful to an anonymous referee for this journal for drawing our attention to this point.
In fact, some have cited epistemic difficulties as a justification for leaving the jus ad bellum criterion of right intention out of the International Law of Armed Conflict (Orend 2006: 47).
Michael Slote (2001) takes a stronger position than Hurka on the moral significance of virtuous motivations: he allows that there can be virtuous motives that do not issue in right acts, but his approach implies that an act is right only if it is virtuously motivated.
For more arguments along these lines, see Kershnar (2013).
It is worth noting that, like unmanned drones, cruise missiles or landmines have the potential to be improved with respect to AI and sensors, which would make them better at discerning targets. Does the fact that such ‘autonomous” landmines would fail to act for the right reasons mean that we are morally required to continue to use ‘dumb’ landmines instead? We concede that our argument entails that there would be at least one serious pro tanto moral objection to deploying autonomous landmines that does not apply to traditional landmines: only autonomous landmines would choose whom to kill, and they would do so without the right reasons. However, this prima facie objection must be weighed against other considerations to arrive at an all-things-considered judgment about their deployment. For instance, traditional landmines are considered problematic by just war theorists because their use often violates the discrimination and non-combatant immunity criterion of jus in bello. Autonomous landmines, which, we are imagining, have the ability to distinguish between legitimate and illegitimate targets would thus be a moral improvement in this regard. We thank an anonymous referee for pressing us on this point.
In this limited sense, all objections to AWS may be contingent in that none of them may justify an absolute prohibition on the deployment of AWS. Still, our objection remains non-contingent insofar as the reason against deploying AWS that we have identified is not dissolved in the presence of countervailing considerations; it is always present, but may simply be outweighed.
Recently, popular attention has been drawn to the possibility that driverless cars will soon replace the human-driven automobile. See, for example, Del-Colle (2013) and Lin (2013b). There are questions about whether driverless cars really would be safer that human-driven cars. But there are several reasons for thinking that driverless cars would be better at driving than humans, in some respects, in the same way that autonomous weapons would be better at soldiering than humans, in some respects. Their faster reaction times and improved calculative abilities are clear. Moreover, driverless cars would not get tired or fatigued, and they could be programmed to drive defensively, as Google’s car is, for example, by automatically avoiding other cars’ blind spots. Nor do driverless cars demonstrate these benefits only when they have the roads to themselves. Google’s driverless cars have already driven over 700,000 miles on public roads occupied almost exclusively by human drivers, and have never been involved in an accident (Anthony 2014). We would expect a human driver to experience about 2.5 accidents in that time (Rosen 2012). According to Bryant Walker Smith at Stanford Law School’s Center for Internet and Society, we are approaching the point at which we can confidently say that Google’s driverless car is significantly safer than human-driven cars (Smith 2012).
We might think that a person surrenders their autonomy in a problematic way if they board a driverless car, but this is not obviously true. Human passengers in driverless cars cannot always choose their routes, but they can choose their destination and can also retake control if they so choose. This is possible, for example, in Google’s driverless car. For this reason, getting into a driverless car surrenders a person’s autonomy less than does getting on a human-piloted airplane or a human-driven bus.
Thank you to an anonymous referee for highlighting this possibility.
We thank an anonymous reviewer on this point.
Again, see Lin (2013a) where this kind of moral dilemma is raised for a driverless car.
This may strike you as equivalent to asking, “What should the avalanche do?” and, like the question about avalanches, may seem confused. It may be more precise to ask, “Which movement of the car would result in a morally better outcome?” or, “What should the programmers of the car have designed the car to do in a case such as this?” Because it is simpler and, we think, coherent, we will continue to speak as if autonomous systems should do certain things.
To use our language above, the advantage to putting driverless cars into use would stem from their abilities to not make as many empirical and practical mistakes that humans do; not their (in)ability to make genuine moral mistakes.
We should be clear that we are not arguing for an absolute prohibition on AWS. We believe there is a strong pro tanto reason against using AWS in contexts where they would be making life or death decisions, but that reason could be outweighed. It could be outweighed if AWS were significantly morally better than human soldiers, for example, because they made fewer moral or practical mistakes. In contexts where this presumption is not outweighed, it could be all-things-considered wrong to deploy autonomous systems.
Sparrow (2007) has made a similar point.
Thanks to an anonymous referee for this journal for pointing out something similar to us.
See, for example, Persson and Savulescu (2010), which argues that we would be obligated to modify humans with AI, if doing so could make us morally better. We go further here and suggest that if AI could on its own be an excellent moral agent, we might be required to outsource all of our moral decisions to it.
Or, instead, should we pity the machine itself? Could it become so overburdened by contemplating the sorrows and tribulations of humanity that it would contrive to have itself destroyed, as did Isaac Asimov’s “Multivac,” a computer designed to solve every human problem (Asimov 1959)?
Some believe that this autonomy is so important that losing one’s autonomy could not be outweighed even by tremendous amounts of other goods. See on this point Valdman (2010).
The authors are indebted to many people for helpful contributions. In particular, we thank Rob Sparrow, David Rodin, Jonathan Parry, Cecile Fabre, Rob Rupert, Andrew Chapman, Leonard Kahn, and two anonymous referees for help on this paper.
- Adams TK (2001) Future warfare and the decline of human decision-making. Parameters: US Army War College Q Winter 2001–2:57–71Google Scholar
- Anthony S (2014) Google’s self-driving car passes 700,000 accident-free miles, can now avoid cyclists, stop at railroad crossings Extreme Tech. http://www.extremetech.com/extreme/181508-googles-self-driving-car-passes-700000-accident-free-miles-can-now-avoid-cyclists-stop-for-trains. Accessed 29 April 2014
- Aquinas T (1920) Summa theologica. 2nd edn. Fathers of the English Dominican ProvinceGoogle Scholar
- Asimov I (1959) Nine tomorrows: tales of the near future. Fawcett, RobinsdaleGoogle Scholar
- Augustine (1887) Contra Faustum Manichaeum. In: Schaff (ed) From Nicene and post-Nicene fathers, first series, vol 4. Christian Literature Publishing Co, BuffaloGoogle Scholar
- Augustine (2004) Letter 189 to Boniface. In Letters 156–210 (II/3) Works of Saint Augustine. New City Press, New YorkGoogle Scholar
- Block N (1978) Troubles for functionalism. Minn Stud Philos Sci 9:261–325Google Scholar
- Borg JS, Sinnott-Armstrong W (2013) Do psychopaths make moral judgments?. In: Kiehl K, Sinnott-Armstrong W (eds) The oxford handbook of psychopathy and law. Oxford, OxfordGoogle Scholar
- Bostrom N (2003) Ethical issues in advanced artificial intelligence. In: Schneider S (ed) Science fiction and philosophy: from time travel to superintelligence. Wiley-Blackwell, 277–286Google Scholar
- Bringsjord S (2007) Offer: one billion dollars for a conscious robot; if you’re honest, you must decline. J Conscious Stud 14(7):28–43Google Scholar
- Brutzman D, Davis D, Lucas GR, McGhee R (2010) Run-time ethics checking for autonomous unmanned vehicles: developing a practical approach. Ethics 9(4):357–383Google Scholar
- Crisp R (2000) Particularizing particularism. In: Hooker B, Little MO (eds) Moral particularism. Oxford, New York, 23–47Google Scholar
- Dancy J (1993) Moral reasons. Wiley-BlackwellGoogle Scholar
- Darwall S (1983) Impartial reason. Cornell, New YorkGoogle Scholar
- Davidson D (1964) Actions, reasons, and causes. J Philos 60(23):685–700Google Scholar
- Davidson D (1978) Intending. In: Yirmiahu (ed) Philosophy and history of action. Springer, p 41–60Google Scholar
- De Greef TE, Arciszewski HF, Neerincx MA (2010) Adaptive automation based on an object-oriented task model: implementation and evaluation in a realistic C2 environment. J Cogn Eng Decis Making 31:152–182Google Scholar
- Del-Colle A (2013) The 12 most important questions about self-driving cars. Popular Mechanics. http://www.popularmechanics.com/cars/news/industry/the-12-most-important-questions-about-self-driving-cars-16016418. Accessed 28 Oct 2013
- Enemark C (2013) Armed drones and the ethics of war: military virtue in a post-heroic age. Routledge, New YorkGoogle Scholar
- Gibbard A (1990) Wise choices, apt feelings. Clarendon, OxfordGoogle Scholar
- Griffin J (1993) How we do ethics now. R Inst Philos Suppl 35:159–177Google Scholar
- Guarini M, Bello P (2012) Robotic warfare: some challenges in moving from noncivilian to civilian theaters. In: Lin P, Abney K, Bekey GA (eds) Robot ethics: the ethical and social implications of robotics. MIT Press, Cambridge, pp 129–144Google Scholar
- Hooker B (2000) Ideal code, real world: a rule-consequentialist theory of morality. Oxford University Press, OxfordGoogle Scholar
- Hurka T (2010) Right act, virtuous motive. In: Battaly HD (ed) Virtue and vice, moral and epistemic. Wiley-Blackwell, 58–72Google Scholar
- Hursthouse R (1995) Applying virtue ethics. Virtues and reasons: 57–75Google Scholar
- Kershnar S (2013) Autonomous weapons pose no moral problems. In: Strawser BJ (ed) Killing by remote control: the ethics of an unmanned military. Oxford University Press, OxfordGoogle Scholar
- Kurzweil R (2000) The age of spiritual machines: when computers exceed human intelligence. PenguinGoogle Scholar
- Kurzweil R (2005) The singularity is near: when humans transcend biology. PenguinGoogle Scholar
- Lin P (2013) The ethics of saving lives with autonomous cars are far murkier than you think. Wired magazine. July 30, 2013. http://www.wired.com/opinion/2013/07/the-surprising-ethics-of-robot-cars/. Accessed 28 Oct 2013
- Lin P (2013) The ethics of autonomous cars. The atlantic. http://www.theatlantic.com/technology/archive/2013/10/the-ethics-of-autonomous-cars/280360/. Accessed 28 Oct 2013
- Little M (2000) Moral generalities revisited. In: Hooker B, Little MO (eds) Moral particularism. Oxford University Press, New York, pp 276–304Google Scholar
- McKeever S, Ridge M (2005) The many moral particularisms. Can J Philos 35(1):83–106Google Scholar
- McNaughton D (1988) Moral vision. Wiley-BlackwellGoogle Scholar
- Mill JS (1863) Utilitarianism. Parker, Son, and Bourn, LondonGoogle Scholar
- Nagel T (1972) War and massacre. Philos Publ Aff 1(2):123–144Google Scholar
- Orend B (2006) The morality of war. Broadview PressGoogle Scholar
- Orend B (2008) War. In: Edward NZ (ed) The stanford encyclopedia of philosophy, (Fall 2008 Edition). URL = <http://plato.stanford.edu/archives/fall2008/entries/war/>
- Quinn W (1993) Morality and action. Cambridge University Press, CambridgeGoogle Scholar
- Rawls J (1971) A theory of justice. Harvard University Press, CambridgeGoogle Scholar
- Roff H (2013) Killing in war: responsibility, liability, and lethal autonomous robots. In: Henschke A, Evans N, Allhoff F (eds) Routledge handbook for ethics and war: just war theory in the 21st century. Routledge, New YorkGoogle Scholar
- Roff H, Momani B (2011) The morality of robotic warfare. National postGoogle Scholar
- Rosen R (2012) Google’s self-driving cars: 300,000 miles logged, not a single accident under computer control. Atlantic. http://www.theatlantic.com/technology/archive/2012/08/googles-self-driving-cars-300-000-miles-logged-not-a-single-accident-under-computer-control/260926/. Accessed 28 Oct 2014
- Scanlon TM (1998) What we owe each other. Harvard University Press, CambridgeGoogle Scholar
- Scheffler S (1992) Human morality. Oxford University Press, New YorkGoogle Scholar
- Schmitt M (2013) Autonomous weapon systems and international humanitarian law: a reply to the critics. Harvard natl. security j. 1–37Google Scholar
- Searle J (1992) The rediscovery of the mind. MIT Press, CambridgeGoogle Scholar
- Setiya K (2007) Reasons without rationalism. Princeton University Press, PrincetonGoogle Scholar
- Smith BW (2012) Driving at perfection. Stanford Law School, Center for Internet and Society. http://cyberlaw.stanford.edu/blog/2012/03/driving-perfection. Accessed 28 Oct 2014
- Vodehnal C (2010) Virtue, practical guidance, and practices. Electronic theses and dissertations. Paper 358. Washington University in St. Louis. http://openscholarship.wustl.edu/etd/358. Accessed 2/12/14
- Walzer M (1977) Just and unjust wars: a moral argument with historical illustrations. Basic Books, New YorkGoogle Scholar